You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/08/10 05:56:28 UTC
[GitHub] [tvm] huajsj opened a new pull request #8702: [Runtime] Pipeline Executor Initial patch.
huajsj opened a new pull request #8702:
URL: https://github.com/apache/tvm/pull/8702
This patch is one of serial patch for PR 7892 splitting. this is initial part the pipeline executor, include the cmake
change and python and c++ interface for pipeline executor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
comaniac commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-897980968
Sorry for the delay review. I'm busy with internal tasks now and will try my best to review the RFC again early next week.
We will be focusing on the RFC and start reviewing the PR with a consent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708863005
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708796993
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
+ to create a pipeline executor.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Build Configuration information.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create a pipeline configuration.
+ string_config[mod_idx] = mconf
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """Pipeline configuration information, this class contains the DAG that expresses
+ the dependency of each module involved by pipeline and the specific parameters
+ of each module build.
+ """
+
+ class Binding:
+ """This class defines the module connections information.
+ The type can only be "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The I/O type of this interface. It can only be "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is an integer such as 0.
+
+ data_type: TensorType
+ The data type of this interface.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child interfaces that depend on this interface.
+ self.bindings = []
+ # Parents interfaces that this interface depend on.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ # Return name of this interface and the name of owner who owns this interface.
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ # If the owner is ModuleWrapper return the owner index, if not return 0.
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use a pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the results computed by the pipeline executor to a caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ # Get all binding information.
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """This is to check whether the DAG containing these input interfaces is acyclic.
+ Parameters
+ ----------
+ start: ModuleWrapper
+ The starting node of the cycle check algorithm.
+
+ inputs: Binding
+ These interfaces are used to connect to each other to build DAG.
+
+ Return
+ ------
+ Return true if there is no cycle in the DAG.
+ """
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Connect the current interface to the destination interface.
+ correct connections as following 1. global input connect to module input,
+ 2. module output connect to global output, 3. module output connect to module input
Review comment:
connect to -> connected to
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704734523
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline runtime.
+ *
+ * This runtime can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \return The type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the graph executor with graph and context.
+ * \param graph_json The execution graph.
+ * \param module The module containing the compiled functions for the host
+ * processor.
+ * \param ctxs The context of the host and devices where graph nodes will be
+ * executed on.
+ * \param lookup_linked_param_func If given, a PackedFunc invoked to lookup linked parameters
+ * by storage_id. If not given, linked parameters are looked-up using an internal implementation,
+ * which is not compatible with RPCModules.
+ */
+ void Init(const Array<tvm::runtime::Module>& modules, const std::string& pipeline_json);
+ /*!
+ * \brief Get member function to front-end.
+ * \param name The name of the function.
+ * \param sptr_to_self The pointer to the module node.
+ * \return The corresponding member function.
+ */
Review comment:
member function -> packed function. Two places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704906413
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """Get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Set index and generating name by index"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Identify if this item is root item, used by DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove self from binding to reduce child in-degree, used by DAG topological sort."""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ """Return item by key"""
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration in dictionary format."""
+
+ # Tpological sort to get correct module order in list.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Get pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build module configuration with pipeline and other parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Do topological sort to get pipeline module order."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return idx for specify mod"""
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704888545
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """Get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Set index and generating name by index"""
Review comment:
generate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704743607
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ class BindingList:
+ """Use to storage Binding list.
Review comment:
This doesn't make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706709877
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706560895
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,270 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # The first model has three output.
+ """
+
+ net1_output1 = relay.add(data, mv1)
+ net1_output2 = relay.subtract(data, mv2)
+ net1_output3 = relay.multiply(data, mv3)
+
+ """
+ # The second model use first model net1_output1 as first input,
+ # here data_net1_output_1 represent net_output1, the second input
+ # of this model is data21.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2_output = relay.add(net2, mv3)
+
+ """
+ # The third model use the second model net2_output as first input
+ # and use the first model net1_output2 as second input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net1_output1, net1_output2, net1_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2_output)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function is used to generate manual pipeline configuration.
+ """
+ mod_config = {}
+ """
+ # The third output is the final output, the second output is for mod3, the first is for mod2
+ # input.
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """
+ # This function is used to trigger runtime error by appling wrong logic connection.
Review comment:
Remove ALL # between """ in this file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697915244
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+ mconfig1 = {}
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ mconfig1["pipeline"] = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = mconfig1
+
+ mconfig2 = {}
+ mconfig2["pipeline"] = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = mconfig2
+
+ mconfig3 = {}
+
+ mconfig3["pipeline"] = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = mconfig3
+ return mod_config
+
+
+def pipeline_module_create(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ pipe_config = pipeline_executor.PipelineModuleConfig([mod1, mod2, mod3])
+
+ # Create pipeline compute input/output and subgraph dependent relation.
+
+ # pipeline compute input "data_0" would get forward to mod1 as input "data_0"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].input("data_0"))
+
+ # pipeline compute input "data_1" would get forward to mod2 as input "data_1"
+ pipe_config.connect(pipe_config.pipe_input("data_1"), pipe_config[mod2].input("data_1"))
+
+ # mod1 output(0) would get forward to mod2 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(0), pipe_config[mod2].input("data_0"))
+
+ # mod1 output(1) would get forward to mod3 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(1), pipe_config[mod3].input("data_0"))
+
+ # mod2 output(0) would get forward to mod3 as input "data_1"
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod3].input("data_1"))
+
+ # mod1 output(2) would get forward as final pipeline compute output(1)
+ pipe_config.connect(pipe_config[mod1].output(2), pipe_config.pipe_output("0"))
+
+ # mod3 output(0) would get forward as final pipeline compute output(2)
+ pipe_config.connect(pipe_config[mod3].output(0), pipe_config.pipe_output("1"))
+ """
+ # print configueration, the expect result like following.
+ #
+ #Inputs
+ # |data_0: mod1:data_0
+ # |data_1: mod2:data_1
+ #
+ #output
+ # |output(1) : mod1.output(2)
+ # |output(2) : mod3.output(0)
+ #
+ #connections
+ # |mod1.output(0)-> mod2.data_0
+ # |mod1.output(1)-> mod3.data_0
+ # |mod2.output(0)-> mod3.data_1
+ """
+
+ print(pipe_config)
+
+ """
+ # connection correctness veify
+ """
+ try:
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod1].input("data_0"))
+ assert 0, f"wrong module connect order check not pass!"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].output(0))
+ assert 0, f"wrong global input connect check not pass!"
+ except:
+ print("connection correctness check pass")
+
+ """
+ # get text format configuration.
+ """
+
+ pconfig = pipe_config.get_config()
+
+ """
+ # check if the configuration match expectation.
+ """
+ assert pconfig == get_manual_conf([mod1, mod2, mod3])
+
+ """
+ # generate configure for build process
+ """
+
+ mod_config = {}
+ mconfig1 = pconfig[mod1]
+ mconfig1["target_host"] = None
+ mconfig1["mod_name"] = "default"
+ mconfig1["build"] = None
+ mconfig1["params"] = None
+ mconfig1["target"] = target[0]
+ mconfig1["dev"] = target[1]
+ mod_config[mod1] = mconfig1
+
+ mconfig2 = pconfig[mod2]
+ mconfig2["target_host"] = None
+ mconfig2["mod_name"] = "default"
+ mconfig2["build"] = None
+ mconfig2["params"] = None
+ mconfig2["target"] = "llvm"
+ mconfig2["dev"] = tvm.cpu(0)
+ mod_config[mod2] = mconfig2
+
+ mconfig3 = pconfig[mod3]
+ mconfig3["target_host"] = None
+ mconfig3["mod_name"] = "default"
+ mconfig3["build"] = None
+ mconfig3["params"] = None
+ mconfig3["target"] = "llvm"
+ mconfig3["dev"] = tvm.cpu(0)
+ mod_config[mod3] = mconfig3
+
+ """
+ # Test build and create pipeline module
+ """
+ with relay.build_config(opt_level=3):
+ pipeline_mods, string_config = pipeline_executor.build_pipeline(mod_config)
+
+ pipeline_module = pipeline_executor.create(pipeline_mods, string_config)
+ return pipeline_module
+
+
+def pipeline(target):
+ module = pipeline_module_create(target)
+ """
+ # Check if pipeline executor create value is valid.
+ """
+ assert module
+
+
+def test_pipeline():
+ if pipeline_executor.pipeline_executor_enabled():
+ target_list = tvm.testing.enabled_targets()
+ for target in target_list:
+ pipeline(target)
Review comment:
this is to address former review comment here https://github.com/apache/tvm/pull/7892#discussion_r625921872, I think the purpose of testing all target is to do correctness check.
for heterogeneous execution here use such way that give fix target for after first module and give dynamic target for the first module to do the work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706561091
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
Review comment:
I commented this before, this is not a factory class. It doesn't create anything new.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707199655
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """Get list of module that represent a subgraph."""
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """The first model has three output."""
+
+ net1_output1 = relay.add(data, mv1)
+ net1_output2 = relay.subtract(data, mv2)
+ net1_output3 = relay.multiply(data, mv3)
+
+ """The second model use first model net1_output1 as first input,
+ here data_net1_output_1 represent net_output1, the second input
+ of this model is data21.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2_output = relay.add(net2, mv3)
+
+ """The third model use the second model net2_output as first input
+ and use the first model net1_output2 as second input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net1_output1, net1_output2, net1_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2_output)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """This function is used to generate manual pipeline configuration."""
+ mod_config = {}
+ """The third output is the final output, the second output is for mod3, the first is for mod2
+ input.
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """This function is used to trigger runtime error by appling wrong logic connection."""
+
+ """Get three pipeline modules here.
Review comment:
For one line comment, just use #, not """. There are many of them in this file, fix all of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707606896
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Check whether the binding settings is correct or not.
+ correct connection are following
+ 1. global input to module input
+ 2. module output to global output
+ 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a cycle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building information.
+ """
+
+ def __init__(self, mod=None):
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
Review comment:
changed into "output_type".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706562221
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
Review comment:
Do not return completely different things from one function.
What is stype
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707208066
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Check whether the binding settings is correct or not.
+ correct connection are following
+ 1. global input to module input
+ 2. module output to global output
+ 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a cycle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building information.
+ """
+
+ def __init__(self, mod=None):
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, interface_type):
+ """Get module interface data type."""
+ if interface_type == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if interface_type == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node, this function used by DAG topological sort."""
Review comment:
Remove "this function"
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Check whether the binding settings is correct or not.
+ correct connection are following
+ 1. global input to module input
+ 2. module output to global output
+ 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a cycle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building information.
+ """
+
+ def __init__(self, mod=None):
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, interface_type):
+ """Get module interface data type."""
+ if interface_type == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if interface_type == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node, this function used by DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree.
+ This function used by DAG topological sort.
Review comment:
is used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704738680
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
Review comment:
I don't think this is IRModule, it should be a runtime Module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697923152
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+ mconfig1 = {}
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ mconfig1["pipeline"] = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = mconfig1
+
+ mconfig2 = {}
+ mconfig2["pipeline"] = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = mconfig2
+
+ mconfig3 = {}
+
+ mconfig3["pipeline"] = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = mconfig3
+ return mod_config
+
+
+def pipeline_module_create(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ pipe_config = pipeline_executor.PipelineModuleConfig([mod1, mod2, mod3])
+
+ # Create pipeline compute input/output and subgraph dependent relation.
+
+ # pipeline compute input "data_0" would get forward to mod1 as input "data_0"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].input("data_0"))
+
+ # pipeline compute input "data_1" would get forward to mod2 as input "data_1"
+ pipe_config.connect(pipe_config.pipe_input("data_1"), pipe_config[mod2].input("data_1"))
+
+ # mod1 output(0) would get forward to mod2 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(0), pipe_config[mod2].input("data_0"))
+
+ # mod1 output(1) would get forward to mod3 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(1), pipe_config[mod3].input("data_0"))
+
+ # mod2 output(0) would get forward to mod3 as input "data_1"
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod3].input("data_1"))
+
+ # mod1 output(2) would get forward as final pipeline compute output(1)
+ pipe_config.connect(pipe_config[mod1].output(2), pipe_config.pipe_output("0"))
+
+ # mod3 output(0) would get forward as final pipeline compute output(2)
+ pipe_config.connect(pipe_config[mod3].output(0), pipe_config.pipe_output("1"))
+ """
+ # print configueration, the expect result like following.
+ #
+ #Inputs
+ # |data_0: mod1:data_0
+ # |data_1: mod2:data_1
+ #
+ #output
+ # |output(1) : mod1.output(2)
+ # |output(2) : mod3.output(0)
+ #
+ #connections
+ # |mod1.output(0)-> mod2.data_0
+ # |mod1.output(1)-> mod3.data_0
+ # |mod2.output(0)-> mod3.data_1
+ """
+
+ print(pipe_config)
+
+ """
+ # connection correctness veify
+ """
+ try:
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod1].input("data_0"))
+ assert 0, f"wrong module connect order check not pass!"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].output(0))
+ assert 0, f"wrong global input connect check not pass!"
+ except:
+ print("connection correctness check pass")
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704887263
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
Review comment:
Please fix and polish ALL error messages in `RuntimeError` above, from L238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-915699650
As I already said, please go through your writing again carefully. I don't want to try to correct all errors, so fixing only the ones I pointed out is not enough. Correct documentation (both in terms of functionality and grammar) would help understand the design better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704890030
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """Get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Set index and generating name by index"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Identify if this item is root item, used by DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove self from binding to reduce child in-degree, used by DAG topological sort."""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ """Return item by key"""
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration in dictionary format."""
+
+ # Tpological sort to get correct module order in list.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Get pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build module configuration with pipeline and other parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Do topological sort to get pipeline module order."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return idx for specify mod"""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return input binding by name"""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return output binding by idx"""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
+ """This is a factory with list of GraphExecutorFactoryModule and Module configurations.
Review comment:
What this is called "factory". It is not creating anything. It just stores existing information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704889337
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """Get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Set index and generating name by index"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Identify if this item is root item, used by DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove self from binding to reduce child in-degree, used by DAG topological sort."""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ """Return item by key"""
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration in dictionary format."""
+
+ # Tpological sort to get correct module order in list.
Review comment:
Topological
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698058090
##########
File path: CMakeLists.txt
##########
@@ -388,6 +388,21 @@ if(GTEST_INCLUDE_DIR AND GTEST_LIB)
include(GoogleTest)
endif()
+if(USE_PIPELINE_EXECUTOR)
+ message(STATUS "Build with Pipeline Executor support...")
+ file(GLOB RUNTIME_PIPELINE_SRCS src/runtime/pipeline/*.cc)
+ list(APPEND RUNTIME_SRCS ${RUNTIME_PIPELINE_SRCS})
+endif(USE_PIPELINE_EXECUTOR)
+
+# Enable ctest if gtest is available
+find_path(GTEST_INCLUDE_DIR gtest/gtest.h)
+find_library(GTEST_LIB gtest "$ENV{GTEST_LIB}")
+if(GTEST_INCLUDE_DIR AND GTEST_LIB)
+ enable_testing()
+ include(CTest)
+ include(GoogleTest)
+endif()
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704824993
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
Review comment:
changed into GraphExecutorFactoryModule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697945139
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+ mconfig1 = {}
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ mconfig1["pipeline"] = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = mconfig1
+
+ mconfig2 = {}
+ mconfig2["pipeline"] = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = mconfig2
+
+ mconfig3 = {}
+
+ mconfig3["pipeline"] = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = mconfig3
+ return mod_config
+
+
+def pipeline_module_create(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ pipe_config = pipeline_executor.PipelineModuleConfig([mod1, mod2, mod3])
+
+ # Create pipeline compute input/output and subgraph dependent relation.
+
+ # pipeline compute input "data_0" would get forward to mod1 as input "data_0"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].input("data_0"))
+
+ # pipeline compute input "data_1" would get forward to mod2 as input "data_1"
+ pipe_config.connect(pipe_config.pipe_input("data_1"), pipe_config[mod2].input("data_1"))
+
+ # mod1 output(0) would get forward to mod2 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(0), pipe_config[mod2].input("data_0"))
+
+ # mod1 output(1) would get forward to mod3 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(1), pipe_config[mod3].input("data_0"))
+
+ # mod2 output(0) would get forward to mod3 as input "data_1"
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod3].input("data_1"))
+
+ # mod1 output(2) would get forward as final pipeline compute output(1)
+ pipe_config.connect(pipe_config[mod1].output(2), pipe_config.pipe_output("0"))
+
+ # mod3 output(0) would get forward as final pipeline compute output(2)
+ pipe_config.connect(pipe_config[mod3].output(0), pipe_config.pipe_output("1"))
+ """
+ # print configueration, the expect result like following.
+ #
+ #Inputs
+ # |data_0: mod1:data_0
+ # |data_1: mod2:data_1
+ #
+ #output
+ # |output(1) : mod1.output(2)
+ # |output(2) : mod3.output(0)
+ #
+ #connections
+ # |mod1.output(0)-> mod2.data_0
+ # |mod1.output(1)-> mod3.data_0
+ # |mod2.output(0)-> mod3.data_1
+ """
+
+ print(pipe_config)
+
+ """
+ # connection correctness veify
+ """
+ try:
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod1].input("data_0"))
+ assert 0, f"wrong module connect order check not pass!"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].output(0))
+ assert 0, f"wrong global input connect check not pass!"
+ except:
+ print("connection correctness check pass")
+
+ """
+ # get text format configuration.
+ """
+
+ pconfig = pipe_config.get_config()
+
+ """
+ # check if the configuration match expectation.
+ """
+ assert pconfig == get_manual_conf([mod1, mod2, mod3])
+
+ """
+ # generate configure for build process
+ """
+
+ mod_config = {}
+ mconfig1 = pconfig[mod1]
+ mconfig1["target_host"] = None
+ mconfig1["mod_name"] = "default"
+ mconfig1["build"] = None
+ mconfig1["params"] = None
+ mconfig1["target"] = target[0]
+ mconfig1["dev"] = target[1]
+ mod_config[mod1] = mconfig1
+
+ mconfig2 = pconfig[mod2]
+ mconfig2["target_host"] = None
+ mconfig2["mod_name"] = "default"
+ mconfig2["build"] = None
+ mconfig2["params"] = None
+ mconfig2["target"] = "llvm"
+ mconfig2["dev"] = tvm.cpu(0)
+ mod_config[mod2] = mconfig2
+
+ mconfig3 = pconfig[mod3]
+ mconfig3["target_host"] = None
+ mconfig3["mod_name"] = "default"
+ mconfig3["build"] = None
+ mconfig3["params"] = None
+ mconfig3["target"] = "llvm"
+ mconfig3["dev"] = tvm.cpu(0)
+ mod_config[mod3] = mconfig3
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704828587
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706880912
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
Review comment:
moved the pipeline executor create logic into PipelineExecutorFactoryModule, now PipelineExecutorFactoryModule response for create and PipelineModule response to expose functions like setinput etc to caller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706702633
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor class use module list and dependency relations of modules as
+ * the parameters and executes these modules on heterogeneous pipeline parallel
Review comment:
changed into "on heterogeneous targets in pipeline parallel"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704901981
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704915691
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,273 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # Net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # Net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # Net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # Set configure
+ """
+
+ """
+ # Third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+ """
+ # Try invalid input/output name exepect runtime error
+ """
+ pipe_error = pipeline_executor.PipelineConfig()
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["output"][9]
+
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["input"]["data_9"]
+
+ """
+ # Try cirle connection , expect runtime error
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r699011713
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707225628
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
Review comment:
not fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707216451
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
Review comment:
Remove `PipelineExecutorFactoryModule` and replace it with a function that does the same thing as ``PipelineExecutorFactoryModule` does in its constructor.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
Review comment:
Remove `PipelineExecutorFactoryModule` and replace it with a function that does the same thing as `PipelineExecutorFactoryModule` does in its constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706876177
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
Review comment:
changed into "this function used by DAG topological sort."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706876104
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
Review comment:
changed into "This function used by DAG topological sort."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704889589
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """Get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Set index and generating name by index"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Identify if this item is root item, used by DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove self from binding to reduce child in-degree, used by DAG topological sort."""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ """Return item by key"""
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration in dictionary format."""
+
+ # Tpological sort to get correct module order in list.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Get pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build module configuration with pipeline and other parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Do topological sort to get pipeline module order."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return idx for specify mod"""
Review comment:
specified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704889250
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """Get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Set index and generating name by index"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Identify if this item is root item, used by DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove self from binding to reduce child in-degree, used by DAG topological sort."""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ """Return item by key"""
Review comment:
Remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r701516534
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704941770
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor take Module list and Dependency relations of Modules as
+ * inputs, then execute modules on heterogeneous hardware target in
+ * pipeline parallism mode to improve processing throughput.
+ *
+ * This executor can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \return The type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the pipeline executor with module Array and json text.
+ * \param modules The module list that use for pipeline build.
+ * \param pipeline_json The configuration for module dependent in pipeline.
Review comment:
changed into "The configuration for module dependencies"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707206004
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
Review comment:
the inputs interfaces -> the input interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706561091
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
Review comment:
I commented this before, this is not a factory class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704891354
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,273 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # Net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # Net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # Net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # Set configure
+ """
+
+ """
+ # Third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+ """
+ # Try invalid input/output name exepect runtime error
Review comment:
typo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704907103
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor take Module list and Dependency relations of Modules as
+ * inputs, then execute modules on heterogeneous hardware target in
+ * pipeline parallism mode to improve processing throughput.
+ *
+ * This executor can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \return The type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the pipeline executor with module Array and json text.
+ * \param modules The module list that use for pipeline build.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704891432
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,273 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # Net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # Net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # Net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # Set configure
+ """
+
+ """
+ # Third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+ """
+ # Try invalid input/output name exepect runtime error
+ """
+ pipe_error = pipeline_executor.PipelineConfig()
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["output"][9]
+
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["input"]["data_9"]
+
+ """
+ # Try cirle connection , expect runtime error
Review comment:
typo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706705453
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
Review comment:
pipeline executor have 2 types input/output, first is global interface that caller use these interface to set input and get output, second is interface of internal subgraph module that build a pipeline from top to down.
after global interface(input) received data, such data would get forward to internal subgraph module interface(input).
when all submodule processed the data follow pipeline, the result would get forward to global output interface, and caller can get such result from global output interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704904792
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706703409
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor class use module list and dependency relations of modules as
+ * the parameters and executes these modules on heterogeneous pipeline parallel
+ * to improve throughput.
+ *
+ * This executor can be acccesibly in various language via
Review comment:
changed to "This executor can be accessed by various language via TVM runtime PackedFunc API"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-912256492
@comaniac, all review comments addressed, please review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706707943
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706880912
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
Review comment:
moved the pipeline executor create logic into PipelineExecutorFactoryModule, now PipelineExecutorFactoryModule response for create and PipelineModule response to expose functions like setinput etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704890893
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor take Module list and Dependency relations of Modules as
+ * inputs, then execute modules on heterogeneous hardware target in
+ * pipeline parallism mode to improve processing throughput.
+ *
+ * This executor can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \return The type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the pipeline executor with module Array and json text.
+ * \param modules The module list that use for pipeline build.
Review comment:
that use -> used
pipeline build -> building a pipeline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707219579
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
Review comment:
uses a pipeline executor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708846035
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
+ to create a pipeline executor.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Build Configuration information.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create a pipeline configuration.
+ string_config[mod_idx] = mconf
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """Pipeline configuration information, this class contains the DAG that expresses
+ the dependency of each module involved by pipeline and the specific parameters
+ of each module build.
Review comment:
changed into "the parameters for building each module."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698845894
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
self.ModuleWrapper(0) is not for first sub module, this object is the global module that represent pipelinemodule and use for exposing global input and output interface/interface binding,
the idea here is to invent a virtual module beside of all sub module to storage the connection relations between pipeline executor input/output and internal sub module for example
```
input:
data_0 : mod0: data_0
data_1: mod1: data_0
output:
ouput(0) : mod0:ouput(0)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704905349
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
Review comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r699606208
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
fixed.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
Review comment:
add the toplogy sort logic, the new API like following
```
pipe_config = pipeline_executor.PipelineConfig()
pipe_config["input"]["data_1"].connect(pipe_config[mod2]["input"]["data_1"])
pipe_config["input"]["data_1"].connect(pipe_config[mod2]["input"]["data_1"])
pipe_config[mod1]["output"][0].connect(pipe_config[mod2]["input"]["data_0"])
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
Review comment:
removed the index check and not rely index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707556317
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
Review comment:
The benefit to make it to be a class is that we can decouple executor maintain logic with executor function export logic, for example after this initial version, we can add a export function to export library and configuration, into PipelineExecutorFactoryModule, the class structure can make such logic be more clear,
at same time PipelineExecutorFactoryModule follow same logic like GraphExecutorFactoryModule that used by GraphExecutor, doing so also can make code logic more consistent. please let me know how you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707836660
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
Review comment:
PipelineExecutorFactoryModule not only do current logic as __init__ look like, after this initial patch we need to add more function for example "export_library" that to export library into disk just like GraphExecutorFactoryModule did,
hence we may need keep it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706703609
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor class use module list and dependency relations of modules as
+ * the parameters and executes these modules on heterogeneous pipeline parallel
+ * to improve throughput.
+ *
+ * This executor can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \Return the type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the pipeline executor with module array and json text.
+ * \param modules The module list that used for building pipeline.
+ * \param pipeline_json The configuration of modules dependencies.
+ */
+ void Init(const Array<tvm::runtime::Module>& modules, const std::string& pipeline_json);
+ /*!
+ * \brief Give frontends an access to packed functions.
+ * \param name The name of the function.
+ * \param sptr_to_self The pointer to the module node.
+ * \return The corresponding packed functions.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r699834273
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
Review comment:
add the toplogy sort logic, the new API like following
```
pipe_config = pipeline_executor.PipelineConfig()
pipe_config["input"]["data_1"].connect(pipe_config[mod2]["input"]["data_1"])
pipe_config["input"]["data_1"].connect(pipe_config[mod2]["input"]["data_1"])
pipe_config[mod1]["output"][0].connect(pipe_config[mod2]["input"]["data_0"])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r699606208
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706561772
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
Review comment:
This comment is not useful. What is global interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706560895
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,270 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # The first model has three output.
+ """
+
+ net1_output1 = relay.add(data, mv1)
+ net1_output2 = relay.subtract(data, mv2)
+ net1_output3 = relay.multiply(data, mv3)
+
+ """
+ # The second model use first model net1_output1 as first input,
+ # here data_net1_output_1 represent net_output1, the second input
+ # of this model is data21.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2_output = relay.add(net2, mv3)
+
+ """
+ # The third model use the second model net2_output as first input
+ # and use the first model net1_output2 as second input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net1_output1, net1_output2, net1_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2_output)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function is used to generate manual pipeline configuration.
+ """
+ mod_config = {}
+ """
+ # The third output is the final output, the second output is for mod3, the first is for mod2
+ # input.
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """
+ # This function is used to trigger runtime error by appling wrong logic connection.
Review comment:
Remove # between """ in this function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704915378
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,273 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # Net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # Net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # Net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706562049
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
Review comment:
Remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704829802
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ class BindingList:
+ """Use to storage Binding list.
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ who own this list, it can be ModuleWrapper or PipelineConfig
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704886349
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
Review comment:
typo
setting is correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706560745
##########
File path: src/runtime/pipeline/pipeline_executor.cc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file pipeline_executor.cc
+ */
+#include "pipeline_executor.h"
+
+namespace tvm {
+namespace runtime {
+
+void PipelineRuntime::Init(const Array<tvm::runtime::Module>& modules,
+ const std::string& pipeline_json) {
+ return;
+}
+
+/* GetFunction can not be pure abstract function, implement an empty function for initial.
Review comment:
for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704738120
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
Review comment:
pipeline_create
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707226292
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Check whether the binding settings is correct or not.
+ correct connection are following
+ 1. global input to module input
+ 2. module output to global output
+ 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a cycle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building information.
+ """
+
+ def __init__(self, mod=None):
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, interface_type):
+ """Get module interface data type."""
+ if interface_type == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if interface_type == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node, this function used by DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree.
+ This function used by DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding to the name."""
Review comment:
according to
I commented this before.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Check whether the binding settings is correct or not.
+ correct connection are following
+ 1. global input to module input
+ 2. module output to global output
+ 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a cycle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building information.
+ """
+
+ def __init__(self, mod=None):
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, interface_type):
+ """Get module interface data type."""
+ if interface_type == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if interface_type == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node, this function used by DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree.
+ This function used by DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding to the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding to the name."""
Review comment:
according to
I commented this before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707219110
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
Review comment:
an integer index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r701525883
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for _, binding in inputs.items():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.dag_acircle_check(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ owner_indx = self.get_owner_indx()
+ bind_owner_indx = binding.get_owner_indx()
+ if owner_indx == bind_owner_indx:
+ raise RuntimeError(f"can not set self as binding.")
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-919658780
Thanks @huajsj for your patience. I left final comments, once they are fixed I'll merge this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704883848
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
Review comment:
The class to store module connection information?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704743328
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ class BindingList:
+ """Use to storage Binding list.
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ who own this list, it can be ModuleWrapper or PipelineConfig
Review comment:
The owner of this list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704830756
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ class BindingList:
+ """Use to storage Binding list.
Review comment:
changed into "Container for bindings(input or output interface)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704915870
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,273 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # Net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # Net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # Net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # Set configure
+ """
+
+ """
+ # Third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+ """
+ # Try invalid input/output name exepect runtime error
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704832810
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706705250
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704886884
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
Review comment:
cycle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704833193
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline runtime.
+ *
+ * This runtime can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \return The type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the graph executor with graph and context.
+ * \param graph_json The execution graph.
+ * \param module The module containing the compiled functions for the host
+ * processor.
+ * \param ctxs The context of the host and devices where graph nodes will be
+ * executed on.
+ * \param lookup_linked_param_func If given, a PackedFunc invoked to lookup linked parameters
+ * by storage_id. If not given, linked parameters are looked-up using an internal implementation,
+ * which is not compatible with RPCModules.
+ */
+ void Init(const Array<tvm::runtime::Module>& modules, const std::string& pipeline_json);
+ /*!
+ * \brief Get member function to front-end.
Review comment:
fixed.
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline runtime.
+ *
+ * This runtime can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \return The type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the graph executor with graph and context.
+ * \param graph_json The execution graph.
+ * \param module The module containing the compiled functions for the host
+ * processor.
+ * \param ctxs The context of the host and devices where graph nodes will be
+ * executed on.
+ * \param lookup_linked_param_func If given, a PackedFunc invoked to lookup linked parameters
+ * by storage_id. If not given, linked parameters are looked-up using an internal implementation,
+ * which is not compatible with RPCModules.
+ */
+ void Init(const Array<tvm::runtime::Module>& modules, const std::string& pipeline_json);
+ /*!
+ * \brief Get member function to front-end.
+ * \param name The name of the function.
+ * \param sptr_to_self The pointer to the module node.
+ * \return The corresponding member function.
+ */
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r699837567
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
Review comment:
removed the index check and not rely index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704888221
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
Review comment:
Remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704736586
##########
File path: src/runtime/pipeline/pipeline_executor.cc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file pipeline_executor.cc
+ */
+#include "pipeline_executor.h"
+
+namespace tvm {
+namespace runtime {
+
+void PipelineRuntime::Init(const Array<tvm::runtime::Module>& modules,
+ const std::string& pipeline_json) {
+ return;
+}
+
+/* GetFunction can not be pure abstract function, implement a empty function for initial.
Review comment:
a empty -> an empty
for initial -> for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r701631601
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <memory>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "../file_utils.h"
+using namespace std;
Review comment:
Remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704901334
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698851127
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
Review comment:
self.mod_name_ get use by graph_executor create and in most case it is "default", so we can not use it here as the name for identification, i would change it into "graph_executor_mod_name" to avoid confuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704643441
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,538 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipeModuleConfig
Review comment:
PipelineExecutorFactoryModule
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,538 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipeModuleConfig
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ for _, binding in self.input_bindings.bindings.items():
+ if binding.parents:
+ return False
+
+ return True
Review comment:
nit:
```suggestion
return all([not b.parents for b in self.input_bindings.bindings.values()])
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,538 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipeModuleConfig
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ for _, binding in self.input_bindings.bindings.items():
+ if binding.parents:
+ return False
+
+ return True
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for _, binding in self.output_bindings.bindings.items():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ class BindingList:
+ """Use to storage Binding list.
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ who own this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Illegal name: {self.binding_type}:{key} cannot find")
+
+ self.bindings[key] = PipelineConfig.ModuleWrapper.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # topology sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for _, interface in self.mod_wrapper[mod].output_bindings.bindings.items():
Review comment:
```suggestion
for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,538 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipeModuleConfig
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ for _, binding in self.input_bindings.bindings.items():
+ if binding.parents:
+ return False
+
+ return True
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for _, binding in self.output_bindings.bindings.items():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ class BindingList:
+ """Use to storage Binding list.
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ who own this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Illegal name: {self.binding_type}:{key} cannot find")
+
+ self.bindings[key] = PipelineConfig.ModuleWrapper.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # topology sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for _, interface in self.mod_wrapper[mod].output_bindings.bindings.items():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += " " + dep_mname + "." + dep_dname
+ connections_dump += " |" + iname + "\n"
+ else:
+ output[dep_dname] = mname + ".output(" + dname + ")"
Review comment:
```suggestion
if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
iname += f" {dep_mname}.{dep_dname}"
connections_dump += f" |{iname\n}"
else:
output[dep_dname] = f"{mname}.output({dname})"
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,538 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipeModuleConfig
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ for _, binding in self.input_bindings.bindings.items():
+ if binding.parents:
+ return False
+
+ return True
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for _, binding in self.output_bindings.bindings.items():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ class BindingList:
+ """Use to storage Binding list.
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ who own this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Illegal name: {self.binding_type}:{key} cannot find")
Review comment:
```suggestion
raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,538 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipeModuleConfig
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ for _, binding in self.input_bindings.bindings.items():
+ if binding.parents:
+ return False
+
+ return True
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for _, binding in self.output_bindings.bindings.items():
Review comment:
```suggestion
for binding in self.output_bindings.bindings.values():
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,538 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipeModuleConfig
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ for _, binding in self.input_bindings.bindings.items():
+ if binding.parents:
+ return False
+
+ return True
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for _, binding in self.output_bindings.bindings.items():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ class BindingList:
+ """Use to storage Binding list.
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ who own this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Illegal name: {self.binding_type}:{key} cannot find")
+
+ self.bindings[key] = PipelineConfig.ModuleWrapper.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # topology sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for _, interface in self.mod_wrapper[mod].output_bindings.bindings.items():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += " " + dep_mname + "." + dep_dname
+ connections_dump += " |" + iname + "\n"
+ else:
+ output[dep_dname] = mname + ".output(" + dname + ")"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += " |output(" + name + ") : " + output[name] + "\n"
Review comment:
```suggestion
output_dump += f" |output("{name}) : {output[name]}\n"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698845894
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
self.ModuleWrapper(0) is not for first sub module, this object is the global module that represent pipelinemodule and use for exposing global input and output interface/interface binding,
the idea here is to invent a virtual module beside of all sub module to storage the connection relations between pipeline executor input/output and internal sub module for example
```
input:
data_0 : mod0: data_0
output:
ouput(0) : mod0:ouput(0)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697945255
##########
File path: src/runtime/pipeline/pipeline_executor.cc
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file pipeline_executor.cc
+ */
+#include "pipeline_executor.h"
+
+namespace tvm {
+namespace runtime {
+
+void SubGraphRuntime::Init(const Array<tvm::runtime::Module>& modules,
+ const std::string& pipeline_json) {
+ return;
+}
+
+PackedFunc SubGraphRuntime::GetFunction(const std::string& name,
+ const ObjectPtr<Object>& sptr_to_self) {
+ return PackedFunc();
Review comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] jroesch commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
jroesch commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-896187735
Is there an RFC for this feature? sorry if it was already posted some where, there is a lot of activity going on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697914291
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+ mconfig1 = {}
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ mconfig1["pipeline"] = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = mconfig1
+
+ mconfig2 = {}
+ mconfig2["pipeline"] = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = mconfig2
+
+ mconfig3 = {}
+
+ mconfig3["pipeline"] = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = mconfig3
+ return mod_config
+
+
+def pipeline_module_create(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ pipe_config = pipeline_executor.PipelineModuleConfig([mod1, mod2, mod3])
+
+ # Create pipeline compute input/output and subgraph dependent relation.
+
+ # pipeline compute input "data_0" would get forward to mod1 as input "data_0"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].input("data_0"))
+
+ # pipeline compute input "data_1" would get forward to mod2 as input "data_1"
+ pipe_config.connect(pipe_config.pipe_input("data_1"), pipe_config[mod2].input("data_1"))
+
+ # mod1 output(0) would get forward to mod2 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(0), pipe_config[mod2].input("data_0"))
+
+ # mod1 output(1) would get forward to mod3 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(1), pipe_config[mod3].input("data_0"))
+
+ # mod2 output(0) would get forward to mod3 as input "data_1"
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod3].input("data_1"))
+
+ # mod1 output(2) would get forward as final pipeline compute output(1)
+ pipe_config.connect(pipe_config[mod1].output(2), pipe_config.pipe_output("0"))
+
+ # mod3 output(0) would get forward as final pipeline compute output(2)
+ pipe_config.connect(pipe_config[mod3].output(0), pipe_config.pipe_output("1"))
+ """
+ # print configueration, the expect result like following.
+ #
+ #Inputs
+ # |data_0: mod1:data_0
+ # |data_1: mod2:data_1
+ #
+ #output
+ # |output(1) : mod1.output(2)
+ # |output(2) : mod3.output(0)
+ #
+ #connections
+ # |mod1.output(0)-> mod2.data_0
+ # |mod1.output(1)-> mod3.data_0
+ # |mod2.output(0)-> mod3.data_1
+ """
+
+ print(pipe_config)
+
+ """
+ # connection correctness veify
+ """
+ try:
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod1].input("data_0"))
+ assert 0, f"wrong module connect order check not pass!"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].output(0))
+ assert 0, f"wrong global input connect check not pass!"
+ except:
+ print("connection correctness check pass")
+
+ """
+ # get text format configuration.
+ """
+
+ pconfig = pipe_config.get_config()
+
+ """
+ # check if the configuration match expectation.
+ """
+ assert pconfig == get_manual_conf([mod1, mod2, mod3])
+
+ """
+ # generate configure for build process
+ """
+
+ mod_config = {}
+ mconfig1 = pconfig[mod1]
+ mconfig1["target_host"] = None
+ mconfig1["mod_name"] = "default"
+ mconfig1["build"] = None
+ mconfig1["params"] = None
+ mconfig1["target"] = target[0]
+ mconfig1["dev"] = target[1]
+ mod_config[mod1] = mconfig1
+
+ mconfig2 = pconfig[mod2]
+ mconfig2["target_host"] = None
+ mconfig2["mod_name"] = "default"
+ mconfig2["build"] = None
+ mconfig2["params"] = None
+ mconfig2["target"] = "llvm"
+ mconfig2["dev"] = tvm.cpu(0)
+ mod_config[mod2] = mconfig2
+
+ mconfig3 = pconfig[mod3]
+ mconfig3["target_host"] = None
+ mconfig3["mod_name"] = "default"
+ mconfig3["build"] = None
+ mconfig3["params"] = None
+ mconfig3["target"] = "llvm"
+ mconfig3["dev"] = tvm.cpu(0)
+ mod_config[mod3] = mconfig3
+
+ """
+ # Test build and create pipeline module
+ """
+ with relay.build_config(opt_level=3):
+ pipeline_mods, string_config = pipeline_executor.build_pipeline(mod_config)
+
+ pipeline_module = pipeline_executor.create(pipeline_mods, string_config)
+ return pipeline_module
+
+
+def pipeline(target):
+ module = pipeline_module_create(target)
+ """
+ # Check if pipeline executor create value is valid.
+ """
+ assert module
Review comment:
fixed
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["lib_name"] = lib_name
+ mconf["json_name"] = json_name
+ mconf["params_name"] = params_name
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return IRModule list and pipeline configuration
+ return mods, string_config
+
+
+def create(pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ submodule = PipelineModule(pipeline_mods, mod_config)
+ return submodule
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ """
+
+ def __init__(self, pipeline_mods, pipeline_config):
+ self.pipeline_mods = pipeline_mods
+ self.mod_config = pipeline_config
+ mods, config = self.graph_executor_create(pipeline_mods, pipeline_config)
+
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : GreaphModule
+ Runtime graph module.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineModuleConfig:
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Instance which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineModuleConfig ---> Pipe Instance ---> Interface(input/output)
+ ---> Module Instance ---> Interface(input/output)
Review comment:
the reason to use one more class to wrap interface is that we need a method to build/find interface by using mod and interface name(for example mod1 input "data_0"), first we can not implement such function inside class interface, second if we implement such function inside PipelineModuleConfig that would like ' PipelineModuleConfig.input(mod1, "data_0") ' , such function is not such straight forward to compare with ' PipelineModuleConfig[mod1].input("data_0")' , hence here I add one additional class for the purpose to make
API usage more readable.
I agree that Instance name is vague, already change it into new name 'Module', please let me know how you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704886434
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
Review comment:
typo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-897994518
thanks @comaniac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r701530077
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for _, binding in inputs.items():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.dag_acircle_check(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ owner_indx = self.get_owner_indx()
+ bind_owner_indx = binding.get_owner_indx()
+ if owner_indx == bind_owner_indx:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if owner_indx != 0 and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if owner_indx != 0 and bind_owner_indx != 0 and binding.io_type == "output":
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if owner_indx != 0 and bind_owner_indx == 0 and binding.io_type == "input":
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if owner_indx == 0 and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if owner_indx == 0 and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(f"Illegal type:binding type is not same!")
+
+ binding.parents.append(self)
+ # Do acircle check after add the in-degree.
+ if not self.dag_acircle_check(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None, index=0):
+ """init class"""
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.set_index_name(index)
+ self.mod = mod
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+ self.target_host_ = None
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_index_name(self, index):
+ """generate name by index and storage index value"""
+ self.index = index
+ self.name = "mod{}".format(str(index))
+
+ def is_root_mod(self):
+ """use by DAG topology sort, identify if this item is root item and in-degree is 0"""
+ for _, binding in self.input_bindings.bindings.items():
+ if binding.parents:
+ return False
+
+ return True
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for _, binding in self.output_bindings.bindings.items():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def set_target_host(self, host):
+ """set target host that use by build function"""
+ self.target_host_ = host
+
+ def set_build_func(self, build_func):
+ """set build funciton that use by build function"""
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ """set params that use by build function"""
+ self.params_ = params
+
+ def set_target(self, target):
+ """set target that use by build function"""
+ self.target_ = target
+
+ def set_dev(self, dev):
+ """set dev that use by build function"""
+ self.dev_ = dev
+
+ class BindingList:
+ """Use to storage Binding list.
+ Parameters
+ ----------
+
+ owner : ModuleWrapper/PipelineConfig
+ who own this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Illegal name: {self.binding_type}:{key} cannot find")
+
+ self.bindings[key] = PipelineConfig.ModuleWrapper.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ def __init__(self):
+ self.last_mod_indx = 0
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # Sort moudles
+ self.dag_topology_sort()
+
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for _, interface in self.mod_wrapper[mod].output_bindings.bindings.items():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += " " + dep_mname + "." + dep_dname
+ connections_dump += " |" + iname + "\n"
+ else:
+ output[dep_dname] = mname + ".output(" + dname + ")"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += " |output(" + name + ") : " + output[name] + "\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ # self.last_mod_indx start from 1 and be initialize value,
+ # the final value for mod index would get generate by function
+ # dag_topology_sort
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706560669
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor class use module list and dependency relations of modules as
+ * the parameters and executes these modules on heterogeneous pipeline parallel
+ * to improve throughput.
+ *
+ * This executor can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \Return the type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the pipeline executor with module array and json text.
+ * \param modules The module list that used for building pipeline.
Review comment:
The module list used for building a pipeline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706561949
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
Review comment:
cycle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706562381
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
Review comment:
What is meant by "and use it in DAG topological sort"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698900038
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
Review comment:
my fault, this variable used by relay.build not graph_executor, yes it can be arbitrary name, I would use self.name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706561432
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
+ """This is a wrapper class which contains GraphExecutorFactoryModule list.
+ and Module configurations.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule.
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
Review comment:
information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r701529923
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704884443
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
Review comment:
what is self name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704939921
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """Get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Set index and generating name by index"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Identify if this item is root item, used by DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove self from binding to reduce child in-degree, used by DAG topological sort."""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ """Return item by key"""
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration in dictionary format."""
+
+ # Tpological sort to get correct module order in list.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Get pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build module configuration with pipeline and other parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Do topological sort to get pipeline module order."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return idx for specify mod"""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return input binding by name"""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return output binding by idx"""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
+ """This is a factory with list of GraphExecutorFactoryModule and Module configurations.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697945743
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707210235
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor class use module list and dependency relations of modules as
+ * the parameters and executes these modules on heterogeneous targets in pipeline
+ * parallel to improve throughput.
Review comment:
"pipeline parallel" is not a word. Use "a pipeline-parallel manner"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707218249
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
Review comment:
includes a IRModule list and a pipeline configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707223292
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Check whether the binding settings is correct or not.
+ correct connection are following
+ 1. global input to module input
+ 2. module output to global output
+ 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a cycle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building information.
+ """
+
+ def __init__(self, mod=None):
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
Review comment:
Why a return type is called "values"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707216971
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
Review comment:
See https://github.com/apache/tvm/pull/8702/files#r707216451
There is no reason this needs to be a class. Replace it with a function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-919731516
@masahi, Thanks for the kindly review, all review comments addressed, please take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708796993
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
+ to create a pipeline executor.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Build Configuration information.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create a pipeline configuration.
+ string_config[mod_idx] = mconf
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """Pipeline configuration information, this class contains the DAG that expresses
+ the dependency of each module involved by pipeline and the specific parameters
+ of each module build.
+ """
+
+ class Binding:
+ """This class defines the module connections information.
+ The type can only be "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The I/O type of this interface. It can only be "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is an integer such as 0.
+
+ data_type: TensorType
+ The data type of this interface.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child interfaces that depend on this interface.
+ self.bindings = []
+ # Parents interfaces that this interface depend on.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ # Return name of this interface and the name of owner who owns this interface.
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ # If the owner is ModuleWrapper return the owner index, if not return 0.
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use a pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the results computed by the pipeline executor to a caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ # Get all binding information.
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """This is to check whether the DAG containing these input interfaces is acyclic.
+ Parameters
+ ----------
+ start: ModuleWrapper
+ The starting node of the cycle check algorithm.
+
+ inputs: Binding
+ These interfaces are used to connect to each other to build DAG.
+
+ Return
+ ------
+ Return true if there is no cycle in the DAG.
+ """
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Connect the current interface to the destination interface.
+ correct connections as following 1. global input connect to module input,
+ 2. module output connect to global output, 3. module output connect to module input
Review comment:
connect to -> connected to, there are three of them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707827054
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
Review comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-918082349
@huajsj See below. You said they were fixed, but they are not.
https://github.com/apache/tvm/pull/8702/files#r706562006
https://github.com/apache/tvm/pull/8702/files#r707197383
https://github.com/apache/tvm/pull/8702#discussion_r706562440
https://github.com/apache/tvm/pull/8702#discussion_r706562444
Before asking a review again, please go through ALL of your writing again and correct as much simple errors as possible (lack of `a` or `the` before a noun, `s` after third-pronoun verb, etc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707825048
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """Get list of module that represent a subgraph."""
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """The first model has three output."""
+
+ net1_output1 = relay.add(data, mv1)
+ net1_output2 = relay.subtract(data, mv2)
+ net1_output3 = relay.multiply(data, mv3)
+
+ """The second model use first model net1_output1 as first input,
+ here data_net1_output_1 represent net_output1, the second input
+ of this model is data21.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2_output = relay.add(net2, mv3)
+
+ """The third model use the second model net2_output as first input
+ and use the first model net1_output2 as second input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net1_output1, net1_output2, net1_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2_output)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """This function is used to generate manual pipeline configuration."""
+ mod_config = {}
+ """The third output is the final output, the second output is for mod3, the first is for mod2
+ input.
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """This function is used to trigger runtime error by appling wrong logic connection."""
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704815854
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-915406454
@comaniac , @masahi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704905607
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """Get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Set index and generating name by index"""
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707219774
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
Review comment:
information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707218724
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
Review comment:
defines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707827054
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
Review comment:
this used in test_pipeline_executor.py, but we can use PipelineModule to replace it.
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707205358
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Check whether the binding settings is correct or not.
Review comment:
setting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707204949
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
Review comment:
results computed by the pipeline executor to a caller
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704882624
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
Review comment:
A class that wraps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704832637
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704815293
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708798413
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
Review comment:
ok makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706561610
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
Review comment:
module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697926981
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["lib_name"] = lib_name
+ mconf["json_name"] = json_name
+ mconf["params_name"] = params_name
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return IRModule list and pipeline configuration
+ return mods, string_config
+
+
+def create(pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ submodule = PipelineModule(pipeline_mods, mod_config)
+ return submodule
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ """
+
+ def __init__(self, pipeline_mods, pipeline_config):
+ self.pipeline_mods = pipeline_mods
+ self.mod_config = pipeline_config
+ mods, config = self.graph_executor_create(pipeline_mods, pipeline_config)
+
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : GreaphModule
+ Runtime graph module.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineModuleConfig:
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Instance which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineModuleConfig ---> Pipe Instance ---> Interface(input/output)
+ ---> Module Instance ---> Interface(input/output)
Review comment:
I would just name it "ModuleWrapper" with module, input and output. So you don't need two dicts for mod and pipe anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706562271
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
Review comment:
What is meant by "and use it in DAG topological sort"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704741975
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
Review comment:
Fix this doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706705216
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
+ """This is a wrapper class which contains GraphExecutorFactoryModule list.
+ and Module configurations.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704891703
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,273 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # Net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # Net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # Net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # Set configure
Review comment:
What is "Set configure"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698848856
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
That makes sense, but this implementation looks confusing, especially module index plays an important role in the current configuration. Since users are not supposed to access the global module, maybe we could just put it separately, like
```
self.root_mod_wrapper = self.ModuleWrapper()
self.mod_wrappers = {}
```
So that we could also make sure `root_mod_wrapper` will not be accessed by users via provided APIs such as `__getitem__`. This also addresses the inconsistent type of the current `self.mod_wrapper: Dict[Union[str, Module], ModuleWrapper]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704902170
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698057179
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
Review comment:
this is old logic, removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704815073
##########
File path: src/runtime/pipeline/pipeline_executor.cc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file pipeline_executor.cc
+ */
+#include "pipeline_executor.h"
+
+namespace tvm {
+namespace runtime {
+
+void PipelineRuntime::Init(const Array<tvm::runtime::Module>& modules,
+ const std::string& pipeline_json) {
+ return;
+}
+
+/* GetFunction can not be pure abstract function, implement a empty function for initial.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698068143
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["lib_name"] = lib_name
+ mconf["json_name"] = json_name
+ mconf["params_name"] = params_name
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return IRModule list and pipeline configuration
+ return mods, string_config
+
+
+def create(pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ submodule = PipelineModule(pipeline_mods, mod_config)
+ return submodule
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ """
+
+ def __init__(self, pipeline_mods, pipeline_config):
+ self.pipeline_mods = pipeline_mods
+ self.mod_config = pipeline_config
+ mods, config = self.graph_executor_create(pipeline_mods, pipeline_config)
+
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : GreaphModule
+ Runtime graph module.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineModuleConfig:
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Instance which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineModuleConfig ---> Pipe Instance ---> Interface(input/output)
+ ---> Module Instance ---> Interface(input/output)
Review comment:
put module into ModuleWrapper means here need a new List to replace dict for mod and pipe tracking, but that seems like not efficient when doing the the PipelineModuleConfig[mod1] search, if the purpose is to reduce dict number into 1 how about just make pipe and mod use same dict?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r700481544
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698716858
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
Review comment:
After reviewing this class carefully, I feel `Binding` might be a more precise term, because this class basically describes how an input/output of this module should come/go.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
Review comment:
1. We don't add `_` suffix in Python and should not. These variables are public after all.
2. `self.itype` -> `self.io_type`.
3. `self.name = str(name)` -> `self.name = str(name) if io_type == "input" else f"output({name})"`.
4. `self.dependent` -> `self.bindings`.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
Review comment:
By your declaration, owner cannot be None.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
Review comment:
In general you need to raise exceptions instead of `assert` in this function. `assert` is usually used for internal error that developers should be aware of. In this case, these errors are triggered by incorrect user behaviors.
Meanwhile, there are two points for the user message:
1. Use capital in the first word.
2. No need to add a period at the end.
```suggestion
def add_binding(self, binding):
"""Check if the binding is legal by looking at the following:
1. Cannot bind to the module itself.
2. Can only bind to later stage modules or global outputs.
3. module output to global output.
"""
owner_indx = self.get_owner_indx()
dep_owner_indx = binding.get_owner_indx()
if owner_indx == dep_owner_indx:
raise RuntimeError(f"Illegal binding: {self.name} cannot bind to the same module")
if owner_indx < dep_owner_indx:
raise Runtimeerror(f"Illegal binding: {self.name} cannot bind to {binding.name} before it")
if dependent.io_type == "output" and dep_owner_indx == 0: # Cannot understand this rule.
raise RuntimeError(f"...")
if owner_indx == 0 and binding.io_type != "input": # Cannot understand this rule either.
raise RuntimeError(f"...")
self.bindings.append(binding)
```
I'm not sure if I changed the conditions correctly as I cannot understand some of them. Please check if the conditions are correct or not.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
Review comment:
```suggestion
self.input_bindings = {}
self.output_bindings = {}
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
Review comment:
Can we just use `self.name`?
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
Review comment:
```suggestion
def __repr__(self):
ret = "{}\n".format(self.name)
for binding in self.bindings:
mname, dname = binding.get_name()
ret += "|- {0}:{1}\n".format(mname, dname)
return name
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
Review comment:
index.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
Review comment:
This is confusing in terms of its name and value.
Suggestion:
```
io_type: str
The type of this binding. It can be either "input" or "output".
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
Review comment:
What's the problem with `mod0`? You need a docstring for this constructor.
```suggestion
self.name = "mod{}".format(str(index) if index else "")
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
Review comment:
Update the docstring accordingly. I would focus on the functionality instead of implementation. For example:
```
The wrapper of each module to be pipelined. The wrapper mainly includes the module itself as well as
the binding that represents the connections of this module's inputs and outputs to other modules.
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
As I mentioned before, I don't think we need to enforce the pipeline inputs to be the inputs of the first module. For example, we should support:
```
data0 -> mod0 -> mod1 -> output
data1 -------------|
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
Review comment:
I feel this is a bit weird. It means when users specify `PipelineModuleConfig([mod0, mod1])`, they already specify the order. However, they still need to specify the connections later on. It would be better to have something like the following:
```
config = PipelineModuleConfig()
config[mod1].input("data0", config.pipeline_input("data0"))
config[mod2].input("data0", config.pipeline_input("data1"))
config[mod2].input("data1", config[mod2].output(0))
config[mod2].output(0, config.pipeline_output(0))
```
When `__getitem__` in pipeline module config sees a new module, it initializes its ModuleWrapper.
The side effect of this approach is that we cannot check if a connection is legal when adding a new connection. However, we can still check that later when exporting the config by leveraging topological sort.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
That makes sense, but this implementation looks confusing, especially module index plays an important role in the current configuration. Since users are not supposed to access the global module, maybe we could just put it separately, like
```
self.root_mod_wrapper = self.ModuleWrapper()
self.mod_wrappers = {}
```
So that we could also make sure `root_mod_wrapper` will not be accessed by users via provided APIs such as `__getitem__`. This also addresses the inconsistent type of the current `self.mod_wrapper: Dict[Union[str, Module], ModuleWrapper]`.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
Review comment:
IIUC, we can use arbitrary name for graph executor and it's not necessary to be "default". Could you check that? If that's not the case then we probably need to discuss with other folks cuz this doesn't make any sense to me.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
Review comment:
This is confusing. If users provide a module list then they would expect the first module is named mod0. However, again it would be better if we don't need to rely on index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704905280
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """Get item by key"""
Review comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704884249
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
Review comment:
What is meant by " The item that self depend". The items that this binding depend on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706560895
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,270 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # The first model has three output.
+ """
+
+ net1_output1 = relay.add(data, mv1)
+ net1_output2 = relay.subtract(data, mv2)
+ net1_output3 = relay.multiply(data, mv3)
+
+ """
+ # The second model use first model net1_output1 as first input,
+ # here data_net1_output_1 represent net_output1, the second input
+ # of this model is data21.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2_output = relay.add(net2, mv3)
+
+ """
+ # The third model use the second model net2_output as first input
+ # and use the first model net1_output2 as second input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net1_output1, net1_output2, net1_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2_output)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function is used to generate manual pipeline configuration.
+ """
+ mod_config = {}
+ """
+ # The third output is the final output, the second output is for mod3, the first is for mod2
+ # input.
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """
+ # This function is used to trigger runtime error by appling wrong logic connection.
Review comment:
Remove # between """ in this file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706560520
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor class use module list and dependency relations of modules as
+ * the parameters and executes these modules on heterogeneous pipeline parallel
Review comment:
in parallel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706876392
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
Review comment:
The return for input and output is same thing that is "data type", the reason we need to check the type is that output of one module will be forwarded to another module as input during pipeline compute, then we need to verify if such forward rule is correct in data type.
I will remove the "input or output" and only mention interface to avoid confuse.
stype is interface type for example "input", changed name into interface_type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704915225
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,273 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # Net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # Net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # Net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698896263
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
Review comment:
indx 0 reserved for the ModuleWraper class that present pipelinemodule, so should not have mod0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708796200
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
+ to create a pipeline executor.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Build Configuration information.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create a pipeline configuration.
+ string_config[mod_idx] = mconf
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """Pipeline configuration information, this class contains the DAG that expresses
+ the dependency of each module involved by pipeline and the specific parameters
Review comment:
involved in a pipeline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707225947
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
Review comment:
not fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698716858
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
Review comment:
After reviewing this class carefully, I feel `Binding` might be a more precise term, because this class basically describes how an input/output of this module should come/go.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
Review comment:
1. We don't add `_` suffix in Python and should not. These variables are public after all.
2. `self.itype` -> `self.io_type`.
3. `self.name = str(name)` -> `self.name = str(name) if io_type == "input" else f"output({name})"`.
4. `self.dependent` -> `self.bindings`.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
Review comment:
By your declaration, owner cannot be None.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
Review comment:
In general you need to raise exceptions instead of `assert` in this function. `assert` is usually used for internal error that developers should be aware of. In this case, these errors are triggered by incorrect user behaviors.
Meanwhile, there are two points for the user message:
1. Use capital in the first word.
2. No need to add a period at the end.
```suggestion
def add_binding(self, binding):
"""Check if the binding is legal by looking at the following:
1. Cannot bind to the module itself.
2. Can only bind to later stage modules or global outputs.
3. module output to global output.
"""
owner_indx = self.get_owner_indx()
dep_owner_indx = binding.get_owner_indx()
if owner_indx == dep_owner_indx:
raise RuntimeError(f"Illegal binding: {self.name} cannot bind to the same module")
if owner_indx < dep_owner_indx:
raise Runtimeerror(f"Illegal binding: {self.name} cannot bind to {binding.name} before it")
if dependent.io_type == "output" and dep_owner_indx == 0: # Cannot understand this rule.
raise RuntimeError(f"...")
if owner_indx == 0 and binding.io_type != "input": # Cannot understand this rule either.
raise RuntimeError(f"...")
self.bindings.append(binding)
```
I'm not sure if I changed the conditions correctly as I cannot understand some of them. Please check if the conditions are correct or not.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
Review comment:
```suggestion
self.input_bindings = {}
self.output_bindings = {}
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
Review comment:
Can we just use `self.name`?
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
Review comment:
```suggestion
def __repr__(self):
ret = "{}\n".format(self.name)
for binding in self.bindings:
mname, dname = binding.get_name()
ret += "|- {0}:{1}\n".format(mname, dname)
return name
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
Review comment:
index.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
Review comment:
This is confusing in terms of its name and value.
Suggestion:
```
io_type: str
The type of this binding. It can be either "input" or "output".
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
Review comment:
What's the problem with `mod0`? You need a docstring for this constructor.
```suggestion
self.name = "mod{}".format(str(index) if index else "")
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
Review comment:
Update the docstring accordingly. I would focus on the functionality instead of implementation. For example:
```
The wrapper of each module to be pipelined. The wrapper mainly includes the module itself as well as
the binding that represents the connections of this module's inputs and outputs to other modules.
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
As I mentioned before, I don't think we need to enforce the pipeline inputs to be the inputs of the first module. For example, we should support:
```
data0 -> mod0 -> mod1 -> output
data1 -------------|
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
Review comment:
I feel this is a bit weird. It means when users specify `PipelineModuleConfig([mod0, mod1])`, they already specify the order. However, they still need to specify the connections later on. It would be better to have something like the following:
```
config = PipelineModuleConfig()
config[mod1].input("data0", config.pipeline_input("data0"))
config[mod2].input("data0", config.pipeline_input("data1"))
config[mod2].input("data1", config[mod2].output(0))
config[mod2].output(0, config.pipeline_output(0))
```
When `__getitem__` in pipeline module config sees a new module, it initializes its ModuleWrapper.
The side effect of this approach is that we cannot check if a connection is legal when adding a new connection. However, we can still check that later when exporting the config by leveraging topological sort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708863161
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
+ to create a pipeline executor.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Build Configuration information.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create a pipeline configuration.
+ string_config[mod_idx] = mconf
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """Pipeline configuration information, this class contains the DAG that expresses
+ the dependency of each module involved by pipeline and the specific parameters
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704905184
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708863805
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
+ to create a pipeline executor.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Build Configuration information.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create a pipeline configuration.
+ string_config[mod_idx] = mconf
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """Pipeline configuration information, this class contains the DAG that expresses
+ the dependency of each module involved by pipeline and the specific parameters
+ of each module build.
+ """
+
+ class Binding:
+ """This class defines the module connections information.
+ The type can only be "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The I/O type of this interface. It can only be "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is an integer such as 0.
+
+ data_type: TensorType
+ The data type of this interface.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child interfaces that depend on this interface.
+ self.bindings = []
+ # Parents interfaces that this interface depend on.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ # Return name of this interface and the name of owner who owns this interface.
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ # If the owner is ModuleWrapper return the owner index, if not return 0.
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use a pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the results computed by the pipeline executor to a caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ # Get all binding information.
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """This is to check whether the DAG containing these input interfaces is acyclic.
+ Parameters
+ ----------
+ start: ModuleWrapper
+ The starting node of the cycle check algorithm.
+
+ inputs: Binding
+ These interfaces are used to connect to each other to build DAG.
+
+ Return
+ ------
+ Return true if there is no cycle in the DAG.
+ """
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Connect the current interface to the destination interface.
+ correct connections as following 1. global input connect to module input,
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi merged pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi merged pull request #8702:
URL: https://github.com/apache/tvm/pull/8702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707217677
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Check whether the binding settings is correct or not.
+ correct connection are following
+ 1. global input to module input
+ 2. module output to global output
+ 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
Review comment:
are same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704892060
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,273 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # Net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # Net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # Net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
Review comment:
use -> is used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697954723
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
Review comment:
for example VTA, when doing build work there are some customize logic like quantize and graph_pack, rpc, for such scenario the use case like following
```python
def vta_build(mod, target, params, target_host, ....):
mod = relay.quantize.quantize(mod, params=params)
mod = graph_pack(mod, ...)
#....
with vta.build_config(opt_level=3, disabled_pass={"AlterOpLayout"}):
lib = relay.build(relay_prog,
target,
params=params,
target_host=ENV.target_host
return lib
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698057236
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["lib_name"] = lib_name
+ mconf["json_name"] = json_name
+ mconf["params_name"] = params_name
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return IRModule list and pipeline configuration
+ return mods, string_config
+
+
+def create(pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ submodule = PipelineModule(pipeline_mods, mod_config)
+ return submodule
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ """
+
+ def __init__(self, pipeline_mods, pipeline_config):
Review comment:
fixed.
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <memory>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "../file_utils.h"
+using namespace std;
+namespace tvm {
+namespace runtime {
+
+/*!
+ * \brief pipeline runtime.
+ *
+ * This runtime can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL SubGraphRuntime : public ModuleNode {
Review comment:
changed into PipelineRuntime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698845894
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
self.ModuleWrapper(0) is not for first sub module, this object is the global module that represent pipelinemodule and use for exposing global input and output interface/interface binding,
the idea here is to invent a virtual module beside of all sub module to storage the connection relations between pipeline executor input/output and internal sub module for example
```
input:
data_0 : mod0: data_0
output:
ouput(0) : mod0:ouput(0)
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
self.ModuleWrapper(0) is not for first sub module, this object is the global module that represent pipelinemodule and use for exposing global input and output interface/interface binding,
the idea here is to invent a virtual module beside of all sub module to storage the connection relations between pipeline executor input/output and internal sub module for example
```
input:
data_0 : mod0: data_0
data_1: mod1: data_0
output:
ouput(0) : mod0:ouput(0)
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
Review comment:
self.mod_name_ get use by graph_executor create and in most case it is "default", so we can not use it here as the name for identification, i would change it into "graph_executor_mod_name" to avoid confuse.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
yes, this more clear, I would apply the change.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
Review comment:
indx 0 reserved for the ModuleWraper class that present pipelinemodule, so should not have mod0.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
Review comment:
my fault, this variable used by relay.build not graph_executor, yes it can be arbitrary name, I would use self.name.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
Review comment:
fixed.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
Review comment:
fixed.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
Review comment:
fixed.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697595674
##########
File path: CMakeLists.txt
##########
@@ -388,6 +388,21 @@ if(GTEST_INCLUDE_DIR AND GTEST_LIB)
include(GoogleTest)
endif()
+if(USE_PIPELINE_EXECUTOR)
+ message(STATUS "Build with Pipeline Executor support...")
+ file(GLOB RUNTIME_PIPELINE_SRCS src/runtime/pipeline/*.cc)
+ list(APPEND RUNTIME_SRCS ${RUNTIME_PIPELINE_SRCS})
+endif(USE_PIPELINE_EXECUTOR)
+
+# Enable ctest if gtest is available
+find_path(GTEST_INCLUDE_DIR gtest/gtest.h)
+find_library(GTEST_LIB gtest "$ENV{GTEST_LIB}")
+if(GTEST_INCLUDE_DIR AND GTEST_LIB)
+ enable_testing()
+ include(CTest)
+ include(GoogleTest)
+endif()
Review comment:
I'm not sure about this. cc @junrushao1994 could you help take a look?
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
Review comment:
```suggestion
"""Pipeline executor that executes a series of modules in a pipeline fashion."""
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
Review comment:
```suggestion
"""check if pipeline executor is enabled.
Return
-------
enable: bool
Return pipeline executor is enabled or not.
```
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <memory>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "../file_utils.h"
+using namespace std;
+namespace tvm {
+namespace runtime {
+
+/*!
+ * \brief pipeline runtime.
+ *
+ * This runtime can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL SubGraphRuntime : public ModuleNode {
Review comment:
Don't use SubgraphRuntime...
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["lib_name"] = lib_name
+ mconf["json_name"] = json_name
+ mconf["params_name"] = params_name
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return IRModule list and pipeline configuration
+ return mods, string_config
+
+
+def create(pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ submodule = PipelineModule(pipeline_mods, mod_config)
+ return submodule
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ """
+
+ def __init__(self, pipeline_mods, pipeline_config):
+ self.pipeline_mods = pipeline_mods
+ self.mod_config = pipeline_config
+ mods, config = self.graph_executor_create(pipeline_mods, pipeline_config)
+
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
Review comment:
Seems not a right description?
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["lib_name"] = lib_name
+ mconf["json_name"] = json_name
+ mconf["params_name"] = params_name
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return IRModule list and pipeline configuration
+ return mods, string_config
+
+
+def create(pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ submodule = PipelineModule(pipeline_mods, mod_config)
+ return submodule
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ """
+
+ def __init__(self, pipeline_mods, pipeline_config):
+ self.pipeline_mods = pipeline_mods
+ self.mod_config = pipeline_config
+ mods, config = self.graph_executor_create(pipeline_mods, pipeline_config)
+
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
Review comment:
```suggestion
assert pipeline_executor_enabled(), "Pipeline executor is not enabled. Please re-build TVM with USE_PIPELINE_EXECUTOR=ON"
pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=False)
```
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+ mconfig1 = {}
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ mconfig1["pipeline"] = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = mconfig1
+
+ mconfig2 = {}
+ mconfig2["pipeline"] = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = mconfig2
+
+ mconfig3 = {}
+
+ mconfig3["pipeline"] = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = mconfig3
+ return mod_config
+
+
+def pipeline_module_create(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ pipe_config = pipeline_executor.PipelineModuleConfig([mod1, mod2, mod3])
+
+ # Create pipeline compute input/output and subgraph dependent relation.
+
+ # pipeline compute input "data_0" would get forward to mod1 as input "data_0"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].input("data_0"))
+
+ # pipeline compute input "data_1" would get forward to mod2 as input "data_1"
+ pipe_config.connect(pipe_config.pipe_input("data_1"), pipe_config[mod2].input("data_1"))
+
+ # mod1 output(0) would get forward to mod2 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(0), pipe_config[mod2].input("data_0"))
+
+ # mod1 output(1) would get forward to mod3 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(1), pipe_config[mod3].input("data_0"))
+
+ # mod2 output(0) would get forward to mod3 as input "data_1"
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod3].input("data_1"))
+
+ # mod1 output(2) would get forward as final pipeline compute output(1)
+ pipe_config.connect(pipe_config[mod1].output(2), pipe_config.pipe_output("0"))
+
+ # mod3 output(0) would get forward as final pipeline compute output(2)
+ pipe_config.connect(pipe_config[mod3].output(0), pipe_config.pipe_output("1"))
+ """
+ # print configueration, the expect result like following.
+ #
+ #Inputs
+ # |data_0: mod1:data_0
+ # |data_1: mod2:data_1
+ #
+ #output
+ # |output(1) : mod1.output(2)
+ # |output(2) : mod3.output(0)
+ #
+ #connections
+ # |mod1.output(0)-> mod2.data_0
+ # |mod1.output(1)-> mod3.data_0
+ # |mod2.output(0)-> mod3.data_1
+ """
+
+ print(pipe_config)
+
+ """
+ # connection correctness veify
+ """
+ try:
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod1].input("data_0"))
+ assert 0, f"wrong module connect order check not pass!"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].output(0))
+ assert 0, f"wrong global input connect check not pass!"
+ except:
+ print("connection correctness check pass")
+
+ """
+ # get text format configuration.
+ """
+
+ pconfig = pipe_config.get_config()
+
+ """
+ # check if the configuration match expectation.
+ """
+ assert pconfig == get_manual_conf([mod1, mod2, mod3])
+
+ """
+ # generate configure for build process
+ """
+
+ mod_config = {}
+ mconfig1 = pconfig[mod1]
+ mconfig1["target_host"] = None
+ mconfig1["mod_name"] = "default"
+ mconfig1["build"] = None
+ mconfig1["params"] = None
+ mconfig1["target"] = target[0]
+ mconfig1["dev"] = target[1]
+ mod_config[mod1] = mconfig1
+
+ mconfig2 = pconfig[mod2]
+ mconfig2["target_host"] = None
+ mconfig2["mod_name"] = "default"
+ mconfig2["build"] = None
+ mconfig2["params"] = None
+ mconfig2["target"] = "llvm"
+ mconfig2["dev"] = tvm.cpu(0)
+ mod_config[mod2] = mconfig2
+
+ mconfig3 = pconfig[mod3]
+ mconfig3["target_host"] = None
+ mconfig3["mod_name"] = "default"
+ mconfig3["build"] = None
+ mconfig3["params"] = None
+ mconfig3["target"] = "llvm"
+ mconfig3["dev"] = tvm.cpu(0)
+ mod_config[mod3] = mconfig3
+
+ """
+ # Test build and create pipeline module
+ """
+ with relay.build_config(opt_level=3):
+ pipeline_mods, string_config = pipeline_executor.build_pipeline(mod_config)
+
+ pipeline_module = pipeline_executor.create(pipeline_mods, string_config)
+ return pipeline_module
+
+
+def pipeline(target):
+ module = pipeline_module_create(target)
+ """
+ # Check if pipeline executor create value is valid.
+ """
+ assert module
Review comment:
Not necessary to be a separate function.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
Review comment:
1. I don't understand this function. Shouldn't it simply accept a pipeline config? In other words, the pipeline config should contain the modules already...
2. We could just name it `build`.
##########
File path: cmake/config.cmake
##########
@@ -105,6 +105,9 @@ set(USE_GRAPH_EXECUTOR ON)
# Whether enable tiny graph executor with CUDA Graph
set(USE_GRAPH_EXECUTOR_CUDA_GRAPH OFF)
+# Whether enable subgraph runtime.
Review comment:
```suggestion
# Whether enable pipeline executor.
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
Review comment:
Why use `enumerate` but not using the index?
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+ mconfig1 = {}
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ mconfig1["pipeline"] = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = mconfig1
+
+ mconfig2 = {}
+ mconfig2["pipeline"] = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = mconfig2
+
+ mconfig3 = {}
+
+ mconfig3["pipeline"] = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = mconfig3
+ return mod_config
+
+
+def pipeline_module_create(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ pipe_config = pipeline_executor.PipelineModuleConfig([mod1, mod2, mod3])
+
+ # Create pipeline compute input/output and subgraph dependent relation.
+
+ # pipeline compute input "data_0" would get forward to mod1 as input "data_0"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].input("data_0"))
+
+ # pipeline compute input "data_1" would get forward to mod2 as input "data_1"
+ pipe_config.connect(pipe_config.pipe_input("data_1"), pipe_config[mod2].input("data_1"))
+
+ # mod1 output(0) would get forward to mod2 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(0), pipe_config[mod2].input("data_0"))
+
+ # mod1 output(1) would get forward to mod3 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(1), pipe_config[mod3].input("data_0"))
+
+ # mod2 output(0) would get forward to mod3 as input "data_1"
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod3].input("data_1"))
+
+ # mod1 output(2) would get forward as final pipeline compute output(1)
+ pipe_config.connect(pipe_config[mod1].output(2), pipe_config.pipe_output("0"))
+
+ # mod3 output(0) would get forward as final pipeline compute output(2)
+ pipe_config.connect(pipe_config[mod3].output(0), pipe_config.pipe_output("1"))
+ """
+ # print configueration, the expect result like following.
+ #
+ #Inputs
+ # |data_0: mod1:data_0
+ # |data_1: mod2:data_1
+ #
+ #output
+ # |output(1) : mod1.output(2)
+ # |output(2) : mod3.output(0)
+ #
+ #connections
+ # |mod1.output(0)-> mod2.data_0
+ # |mod1.output(1)-> mod3.data_0
+ # |mod2.output(0)-> mod3.data_1
+ """
+
+ print(pipe_config)
Review comment:
Remove print in the test.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
Review comment:
Can't understand why we need this logic.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
Review comment:
```suggestion
return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["lib_name"] = lib_name
+ mconf["json_name"] = json_name
+ mconf["params_name"] = params_name
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return IRModule list and pipeline configuration
+ return mods, string_config
Review comment:
We should not expose them separately to users. Users should just see a single object returned by this function. In other words, this entire function should be wrapped in `create` so that we could just have
```
with relay.build_config(opt_level=3):
pipeline_module = pipeline_executor.create(mod_config)
return pipeline_module
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["lib_name"] = lib_name
+ mconf["json_name"] = json_name
+ mconf["params_name"] = params_name
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return IRModule list and pipeline configuration
+ return mods, string_config
+
+
+def create(pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ submodule = PipelineModule(pipeline_mods, mod_config)
+ return submodule
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ """
+
+ def __init__(self, pipeline_mods, pipeline_config):
Review comment:
Again, can we just put modules into pipeline config so that we don't need to ask users to provide two things separately?
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+ mconfig1 = {}
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ mconfig1["pipeline"] = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = mconfig1
+
+ mconfig2 = {}
+ mconfig2["pipeline"] = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = mconfig2
+
+ mconfig3 = {}
+
+ mconfig3["pipeline"] = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = mconfig3
+ return mod_config
+
+
+def pipeline_module_create(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ pipe_config = pipeline_executor.PipelineModuleConfig([mod1, mod2, mod3])
+
+ # Create pipeline compute input/output and subgraph dependent relation.
+
+ # pipeline compute input "data_0" would get forward to mod1 as input "data_0"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].input("data_0"))
+
+ # pipeline compute input "data_1" would get forward to mod2 as input "data_1"
+ pipe_config.connect(pipe_config.pipe_input("data_1"), pipe_config[mod2].input("data_1"))
+
+ # mod1 output(0) would get forward to mod2 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(0), pipe_config[mod2].input("data_0"))
+
+ # mod1 output(1) would get forward to mod3 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(1), pipe_config[mod3].input("data_0"))
+
+ # mod2 output(0) would get forward to mod3 as input "data_1"
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod3].input("data_1"))
+
+ # mod1 output(2) would get forward as final pipeline compute output(1)
+ pipe_config.connect(pipe_config[mod1].output(2), pipe_config.pipe_output("0"))
+
+ # mod3 output(0) would get forward as final pipeline compute output(2)
+ pipe_config.connect(pipe_config[mod3].output(0), pipe_config.pipe_output("1"))
+ """
+ # print configueration, the expect result like following.
+ #
+ #Inputs
+ # |data_0: mod1:data_0
+ # |data_1: mod2:data_1
+ #
+ #output
+ # |output(1) : mod1.output(2)
+ # |output(2) : mod3.output(0)
+ #
+ #connections
+ # |mod1.output(0)-> mod2.data_0
+ # |mod1.output(1)-> mod3.data_0
+ # |mod2.output(0)-> mod3.data_1
+ """
+
+ print(pipe_config)
+
+ """
+ # connection correctness veify
+ """
+ try:
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod1].input("data_0"))
+ assert 0, f"wrong module connect order check not pass!"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].output(0))
+ assert 0, f"wrong global input connect check not pass!"
+ except:
+ print("connection correctness check pass")
+
+ """
+ # get text format configuration.
+ """
+
+ pconfig = pipe_config.get_config()
+
+ """
+ # check if the configuration match expectation.
+ """
+ assert pconfig == get_manual_conf([mod1, mod2, mod3])
+
+ """
+ # generate configure for build process
+ """
+
+ mod_config = {}
+ mconfig1 = pconfig[mod1]
+ mconfig1["target_host"] = None
+ mconfig1["mod_name"] = "default"
+ mconfig1["build"] = None
+ mconfig1["params"] = None
+ mconfig1["target"] = target[0]
+ mconfig1["dev"] = target[1]
+ mod_config[mod1] = mconfig1
+
+ mconfig2 = pconfig[mod2]
+ mconfig2["target_host"] = None
+ mconfig2["mod_name"] = "default"
+ mconfig2["build"] = None
+ mconfig2["params"] = None
+ mconfig2["target"] = "llvm"
+ mconfig2["dev"] = tvm.cpu(0)
+ mod_config[mod2] = mconfig2
+
+ mconfig3 = pconfig[mod3]
+ mconfig3["target_host"] = None
+ mconfig3["mod_name"] = "default"
+ mconfig3["build"] = None
+ mconfig3["params"] = None
+ mconfig3["target"] = "llvm"
+ mconfig3["dev"] = tvm.cpu(0)
+ mod_config[mod3] = mconfig3
Review comment:
This programming model is unacceptable. i.e., it's improper to ask users to write this.
You might need to also put them into pipe_config with user friendly APIs. Maybe something like
```
pipe_config = pipeline_executor.PipelineModuleConfig()
pipe_config.add_module(mod1, ...) # The "..." here should align the interface of relay.build.
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["lib_name"] = lib_name
+ mconf["json_name"] = json_name
+ mconf["params_name"] = params_name
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return IRModule list and pipeline configuration
+ return mods, string_config
+
+
+def create(pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ submodule = PipelineModule(pipeline_mods, mod_config)
+ return submodule
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ """
+
+ def __init__(self, pipeline_mods, pipeline_config):
+ self.pipeline_mods = pipeline_mods
+ self.mod_config = pipeline_config
+ mods, config = self.graph_executor_create(pipeline_mods, pipeline_config)
+
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : GreaphModule
+ Runtime graph module.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineModuleConfig:
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Instance which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineModuleConfig ---> Pipe Instance ---> Interface(input/output)
+ ---> Module Instance ---> Interface(input/output)
Review comment:
- It looks confusing. Is it possible to not have pipe instance and module instance?
- The name `Instance` is too vague.
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+ mconfig1 = {}
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ mconfig1["pipeline"] = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = mconfig1
+
+ mconfig2 = {}
+ mconfig2["pipeline"] = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = mconfig2
+
+ mconfig3 = {}
+
+ mconfig3["pipeline"] = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = mconfig3
+ return mod_config
+
+
+def pipeline_module_create(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ pipe_config = pipeline_executor.PipelineModuleConfig([mod1, mod2, mod3])
+
+ # Create pipeline compute input/output and subgraph dependent relation.
+
+ # pipeline compute input "data_0" would get forward to mod1 as input "data_0"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].input("data_0"))
+
+ # pipeline compute input "data_1" would get forward to mod2 as input "data_1"
+ pipe_config.connect(pipe_config.pipe_input("data_1"), pipe_config[mod2].input("data_1"))
+
+ # mod1 output(0) would get forward to mod2 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(0), pipe_config[mod2].input("data_0"))
+
+ # mod1 output(1) would get forward to mod3 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(1), pipe_config[mod3].input("data_0"))
+
+ # mod2 output(0) would get forward to mod3 as input "data_1"
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod3].input("data_1"))
+
+ # mod1 output(2) would get forward as final pipeline compute output(1)
+ pipe_config.connect(pipe_config[mod1].output(2), pipe_config.pipe_output("0"))
+
+ # mod3 output(0) would get forward as final pipeline compute output(2)
+ pipe_config.connect(pipe_config[mod3].output(0), pipe_config.pipe_output("1"))
+ """
+ # print configueration, the expect result like following.
+ #
+ #Inputs
+ # |data_0: mod1:data_0
+ # |data_1: mod2:data_1
+ #
+ #output
+ # |output(1) : mod1.output(2)
+ # |output(2) : mod3.output(0)
+ #
+ #connections
+ # |mod1.output(0)-> mod2.data_0
+ # |mod1.output(1)-> mod3.data_0
+ # |mod2.output(0)-> mod3.data_1
+ """
+
+ print(pipe_config)
+
+ """
+ # connection correctness veify
+ """
+ try:
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod1].input("data_0"))
+ assert 0, f"wrong module connect order check not pass!"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].output(0))
+ assert 0, f"wrong global input connect check not pass!"
+ except:
+ print("connection correctness check pass")
+
+ """
+ # get text format configuration.
+ """
+
+ pconfig = pipe_config.get_config()
+
+ """
+ # check if the configuration match expectation.
+ """
+ assert pconfig == get_manual_conf([mod1, mod2, mod3])
+
+ """
+ # generate configure for build process
+ """
+
+ mod_config = {}
+ mconfig1 = pconfig[mod1]
+ mconfig1["target_host"] = None
+ mconfig1["mod_name"] = "default"
+ mconfig1["build"] = None
+ mconfig1["params"] = None
+ mconfig1["target"] = target[0]
+ mconfig1["dev"] = target[1]
+ mod_config[mod1] = mconfig1
+
+ mconfig2 = pconfig[mod2]
+ mconfig2["target_host"] = None
+ mconfig2["mod_name"] = "default"
+ mconfig2["build"] = None
+ mconfig2["params"] = None
+ mconfig2["target"] = "llvm"
+ mconfig2["dev"] = tvm.cpu(0)
+ mod_config[mod2] = mconfig2
+
+ mconfig3 = pconfig[mod3]
+ mconfig3["target_host"] = None
+ mconfig3["mod_name"] = "default"
+ mconfig3["build"] = None
+ mconfig3["params"] = None
+ mconfig3["target"] = "llvm"
+ mconfig3["dev"] = tvm.cpu(0)
+ mod_config[mod3] = mconfig3
+
+ """
+ # Test build and create pipeline module
+ """
+ with relay.build_config(opt_level=3):
+ pipeline_mods, string_config = pipeline_executor.build_pipeline(mod_config)
+
+ pipeline_module = pipeline_executor.create(pipeline_mods, string_config)
+ return pipeline_module
+
+
+def pipeline(target):
+ module = pipeline_module_create(target)
+ """
+ # Check if pipeline executor create value is valid.
+ """
+ assert module
+
+
+def test_pipeline():
+ if pipeline_executor.pipeline_executor_enabled():
+ target_list = tvm.testing.enabled_targets()
+ for target in target_list:
+ pipeline(target)
Review comment:
Do you really need to test all targets? If you want to test heterogeneous execution, then you could just fix the target for each module.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
Review comment:
What could be the self-defined build function other than relay.build?
##########
File path: src/runtime/pipeline/pipeline_executor.cc
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file pipeline_executor.cc
+ */
+#include "pipeline_executor.h"
+
+namespace tvm {
+namespace runtime {
+
+void SubGraphRuntime::Init(const Array<tvm::runtime::Module>& modules,
+ const std::string& pipeline_json) {
+ return;
+}
+
+PackedFunc SubGraphRuntime::GetFunction(const std::string& name,
+ const ObjectPtr<Object>& sptr_to_self) {
+ return PackedFunc();
Review comment:
Why?
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+ mconfig1 = {}
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ mconfig1["pipeline"] = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = mconfig1
+
+ mconfig2 = {}
+ mconfig2["pipeline"] = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = mconfig2
+
+ mconfig3 = {}
+
+ mconfig3["pipeline"] = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = mconfig3
+ return mod_config
+
+
+def pipeline_module_create(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ pipe_config = pipeline_executor.PipelineModuleConfig([mod1, mod2, mod3])
+
+ # Create pipeline compute input/output and subgraph dependent relation.
+
+ # pipeline compute input "data_0" would get forward to mod1 as input "data_0"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].input("data_0"))
+
+ # pipeline compute input "data_1" would get forward to mod2 as input "data_1"
+ pipe_config.connect(pipe_config.pipe_input("data_1"), pipe_config[mod2].input("data_1"))
+
+ # mod1 output(0) would get forward to mod2 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(0), pipe_config[mod2].input("data_0"))
+
+ # mod1 output(1) would get forward to mod3 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(1), pipe_config[mod3].input("data_0"))
+
+ # mod2 output(0) would get forward to mod3 as input "data_1"
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod3].input("data_1"))
+
+ # mod1 output(2) would get forward as final pipeline compute output(1)
+ pipe_config.connect(pipe_config[mod1].output(2), pipe_config.pipe_output("0"))
+
+ # mod3 output(0) would get forward as final pipeline compute output(2)
+ pipe_config.connect(pipe_config[mod3].output(0), pipe_config.pipe_output("1"))
+ """
+ # print configueration, the expect result like following.
+ #
+ #Inputs
+ # |data_0: mod1:data_0
+ # |data_1: mod2:data_1
+ #
+ #output
+ # |output(1) : mod1.output(2)
+ # |output(2) : mod3.output(0)
+ #
+ #connections
+ # |mod1.output(0)-> mod2.data_0
+ # |mod1.output(1)-> mod3.data_0
+ # |mod2.output(0)-> mod3.data_1
+ """
+
+ print(pipe_config)
+
+ """
+ # connection correctness veify
+ """
+ try:
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod1].input("data_0"))
+ assert 0, f"wrong module connect order check not pass!"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].output(0))
+ assert 0, f"wrong global input connect check not pass!"
+ except:
+ print("connection correctness check pass")
Review comment:
Don't test in this way...if you expect to have exception, using the following
```python
with pytest.raises(RuntimeError):
pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod1].input("data_0"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704686221
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,538 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipeModuleConfig
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704901616
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697945477
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["lib_name"] = lib_name
+ mconf["json_name"] = json_name
+ mconf["params_name"] = params_name
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return IRModule list and pipeline configuration
+ return mods, string_config
+
+
+def create(pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ submodule = PipelineModule(pipeline_mods, mod_config)
+ return submodule
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ """
+
+ def __init__(self, pipeline_mods, pipeline_config):
+ self.pipeline_mods = pipeline_mods
+ self.mod_config = pipeline_config
+ mods, config = self.graph_executor_create(pipeline_mods, pipeline_config)
+
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create a pipeline runtime executor.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704906735
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706560529
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor class use module list and dependency relations of modules as
Review comment:
uses a module list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704915426
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,273 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # Net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # Net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # Net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # Set configure
Review comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706705453
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
Review comment:
changed into following comment
"""The global interface is the interface visible to the caller which use pipeline executor, the global input interface is responsible for passing parameters to the internal module interface, and the global output interface is responsible for outputting the pipeline result to the caller."""
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708795562
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
Review comment:
ok makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707213422
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
Review comment:
Is this actually used? I suggest remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707203410
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
Review comment:
This class and `create` function above seem too trivial. I suggest remove them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707207644
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Check whether the binding settings is correct or not.
+ correct connection are following
+ 1. global input to module input
+ 2. module output to global output
+ 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a cycle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building information.
Review comment:
information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707214786
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
Review comment:
The doc is outdated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707824767
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """Get list of module that represent a subgraph."""
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """The first model has three output."""
+
+ net1_output1 = relay.add(data, mv1)
+ net1_output2 = relay.subtract(data, mv2)
+ net1_output3 = relay.multiply(data, mv3)
+
+ """The second model use first model net1_output1 as first input,
+ here data_net1_output_1 represent net_output1, the second input
+ of this model is data21.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2_output = relay.add(net2, mv3)
+
+ """The third model use the second model net2_output as first input
+ and use the first model net1_output2 as second input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net1_output1, net1_output2, net1_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2_output)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """This function is used to generate manual pipeline configuration."""
+ mod_config = {}
+ """The third output is the final output, the second output is for mod3, the first is for mod2
+ input.
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """This function is used to trigger runtime error by appling wrong logic connection."""
+
+ """Get three pipeline modules here.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r705666642
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697946280
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
Review comment:
removed enumerate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704903960
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-916417269
thanks @masahi, all review comments get addressed, please review again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704735999
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline runtime.
+ *
+ * This runtime can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
Review comment:
Please explain how this runtime is different from other runtimes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704904981
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
Review comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706703634
##########
File path: src/runtime/pipeline/pipeline_executor.cc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file pipeline_executor.cc
+ */
+#include "pipeline_executor.h"
+
+namespace tvm {
+namespace runtime {
+
+void PipelineRuntime::Init(const Array<tvm::runtime::Module>& modules,
+ const std::string& pipeline_json) {
+ return;
+}
+
+/* GetFunction can not be pure abstract function, implement an empty function for initial.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706560545
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor class use module list and dependency relations of modules as
+ * the parameters and executes these modules on heterogeneous pipeline parallel
+ * to improve throughput.
+ *
+ * This executor can be acccesibly in various language via
Review comment:
accessible by
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706561844
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
Review comment:
contains
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706561903
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
Review comment:
Remove # between """
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704905817
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """Get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Set index and generating name by index"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Identify if this item is root item, used by DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove self from binding to reduce child in-degree, used by DAG topological sort."""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704946822
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
Review comment:
changed into "Return this interface name and name of owner who own this interface."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706561715
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
Review comment:
owns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706702680
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor class use module list and dependency relations of modules as
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704742629
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ class BindingList:
+ """Use to storage Binding list.
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ who own this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.ModuleWrapper.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # topology sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration in dictionary format."""
+
+ # topology sort to get correct module order in list.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # get pipeline configure
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # build module configuration with pipeline and other parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Do topology sort to get pipeline module order."""
Review comment:
topological sort
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706562444
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
Review comment:
according to
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704737300
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
Review comment:
remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706561475
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
+ """This is a wrapper class which contains GraphExecutorFactoryModule list.
+ and Module configurations.
Review comment:
module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706875829
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r701522030
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for _, binding in inputs.items():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.dag_acircle_check(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ owner_indx = self.get_owner_indx()
+ bind_owner_indx = binding.get_owner_indx()
+ if owner_indx == bind_owner_indx:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if owner_indx != 0 and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if owner_indx != 0 and bind_owner_indx != 0 and binding.io_type == "output":
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if owner_indx != 0 and bind_owner_indx == 0 and binding.io_type == "input":
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if owner_indx == 0 and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if owner_indx == 0 and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704744904
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
Review comment:
Used by
to remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704745040
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
Review comment:
Used by
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704735393
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline runtime.
+ *
+ * This runtime can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \return The type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the graph executor with graph and context.
+ * \param graph_json The execution graph.
+ * \param module The module containing the compiled functions for the host
+ * processor.
+ * \param ctxs The context of the host and devices where graph nodes will be
+ * executed on.
+ * \param lookup_linked_param_func If given, a PackedFunc invoked to lookup linked parameters
+ * by storage_id. If not given, linked parameters are looked-up using an internal implementation,
+ * which is not compatible with RPCModules.
+ */
+ void Init(const Array<tvm::runtime::Module>& modules, const std::string& pipeline_json);
+ /*!
+ * \brief Get member function to front-end.
Review comment:
Give frontends an access to packed functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704883612
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
Review comment:
owns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704887844
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
Review comment:
Remove this comment as it is not helpful.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
Review comment:
Remove this comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706708057
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704828739
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
Review comment:
fixed, put all 3 subclass in same level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r697945164
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,256 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+ mconfig1 = {}
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ mconfig1["pipeline"] = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = mconfig1
+
+ mconfig2 = {}
+ mconfig2["pipeline"] = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = mconfig2
+
+ mconfig3 = {}
+
+ mconfig3["pipeline"] = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = mconfig3
+ return mod_config
+
+
+def pipeline_module_create(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ pipe_config = pipeline_executor.PipelineModuleConfig([mod1, mod2, mod3])
+
+ # Create pipeline compute input/output and subgraph dependent relation.
+
+ # pipeline compute input "data_0" would get forward to mod1 as input "data_0"
+ pipe_config.connect(pipe_config.pipe_input("data_0"), pipe_config[mod1].input("data_0"))
+
+ # pipeline compute input "data_1" would get forward to mod2 as input "data_1"
+ pipe_config.connect(pipe_config.pipe_input("data_1"), pipe_config[mod2].input("data_1"))
+
+ # mod1 output(0) would get forward to mod2 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(0), pipe_config[mod2].input("data_0"))
+
+ # mod1 output(1) would get forward to mod3 as input "data_0"
+ pipe_config.connect(pipe_config[mod1].output(1), pipe_config[mod3].input("data_0"))
+
+ # mod2 output(0) would get forward to mod3 as input "data_1"
+ pipe_config.connect(pipe_config[mod2].output(0), pipe_config[mod3].input("data_1"))
+
+ # mod1 output(2) would get forward as final pipeline compute output(1)
+ pipe_config.connect(pipe_config[mod1].output(2), pipe_config.pipe_output("0"))
+
+ # mod3 output(0) would get forward as final pipeline compute output(2)
+ pipe_config.connect(pipe_config[mod3].output(0), pipe_config.pipe_output("1"))
+ """
+ # print configueration, the expect result like following.
+ #
+ #Inputs
+ # |data_0: mod1:data_0
+ # |data_1: mod2:data_1
+ #
+ #output
+ # |output(1) : mod1.output(2)
+ # |output(2) : mod3.output(0)
+ #
+ #connections
+ # |mod1.output(0)-> mod2.data_0
+ # |mod1.output(1)-> mod3.data_0
+ # |mod2.output(0)-> mod3.data_1
+ """
+
+ print(pipe_config)
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r699020077
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706708597
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r701563498
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704902602
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698057295
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,352 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes pipeline containing TVM PackedFunc."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor enabled.
+ Return
+ ------
+ enable: bool
+ return pipeline executor get enabled or not
+ """
+ pipeline_enabled = False
+ try:
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create")
+ assert pipelinecreate
+ pipeline_enabled = True
+ except ValueError:
+ print("pipeline executor not enabled!")
+
+ return pipeline_enabled
+
+
+def build_pipeline(mod_n_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ # init lib_name and json_name params with empty
+ lib_name = ""
+ json_name = ""
+ params_name = ""
+ # Get module configuration
+ assert "pipeline" in mod_config and "mod_indx" in mod_config["pipeline"]
+ # Get module index in pipeline configuration
+ mconf = mod_config["pipeline"].copy()
+ # Get mod device config
+ dev = mod_config["dev"]
+ mod_indx = mconf["mod_indx"] - 1
+ target = mod_config["target"]
+ assert mod_indx < config_len
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["lib_name"] = lib_name
+ mconf["json_name"] = json_name
+ mconf["params_name"] = params_name
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return IRModule list and pipeline configuration
+ return mods, string_config
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r701467431
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
Review comment:
Update docstring
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,276 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ pipe_config1 = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def pipeline(target):
Review comment:
You may consider inline this function directly to `test_pipeline` as it is only used by one function.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for _, binding in inputs.items():
Review comment:
```suggestion
for binding in inputs.values():
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
Review comment:
This shouldn't be a "config". It's weird to see something like `mod_config = build(pipe_config)`.
This should derive from `ExecutorFactoryModule` like `GraphExecutorFactoryModule`, so maybe call it `PipelineExecutorFactoryModule`.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
Review comment:
```suggestion
"""The class that use to storage module connection information.
The binding can be either "input" or "output".
Parameters
----------
owner : ModuleWrapper
The class that own this interface, in such class there are
Module information like index, module name
io_type : str
The type of this binding. It can be either "input" or "output".
name : str/integer
Binding name, for input it is string such as "data0";
for output it is the index integer such as 0.
```
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,276 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ pipe_config1 = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def pipeline(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ # Runtime error check
+ pipe_config_check(mod1, mod2, mod3)
Review comment:
Put this to a separate test. For example, you can have
```python
def test_pipeline_config():
(mod1, mod2, mod3), dshape = get_mannual_mod()
# Inline "pipe_config_check" here
```
This check should be target independent so it should not be tested many times. Putting it to a separate test can also help locate the problem when test failure.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for _, binding in inputs.items():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.dag_acircle_check(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ owner_indx = self.get_owner_indx()
+ bind_owner_indx = binding.get_owner_indx()
+ if owner_indx == bind_owner_indx:
+ raise RuntimeError(f"can not set self as binding.")
Review comment:
By getting rid of the index, you should implement this check like
```
if self.io_owner == binding.io_owner:
raise RuntimeError(f"Can not set self as binding.")
```
Then you need to implement `ModuleWrapper.__eq__(self, other)` to support the `==` operator.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for _, binding in inputs.items():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.dag_acircle_check(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ owner_indx = self.get_owner_indx()
+ bind_owner_indx = binding.get_owner_indx()
+ if owner_indx == bind_owner_indx:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if owner_indx != 0 and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if owner_indx != 0 and bind_owner_indx != 0 and binding.io_type == "output":
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if owner_indx != 0 and bind_owner_indx == 0 and binding.io_type == "input":
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if owner_indx == 0 and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if owner_indx == 0 and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
Review comment:
This can also be `if not self.is_global_interface()`
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
Review comment:
The word "acircle" seems not that common. Maybe just "check_dag_acyclic" and return True if it is valid (no circle)?
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for _, binding in inputs.items():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.dag_acircle_check(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ owner_indx = self.get_owner_indx()
+ bind_owner_indx = binding.get_owner_indx()
+ if owner_indx == bind_owner_indx:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if owner_indx != 0 and self.io_type == "input":
Review comment:
Again, all `owner_idx != 0` can just be `not self.is_global_interface()`.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
Review comment:
I still feel use 0 to indicate global interface is not clear enough. I read through the use cases, you only use it for `index != 0`, so it should be fine to just make this function to be
```
def is_global_interface(self):
return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
```
btw, please change all `xx_indx` to `xx_index` or `xx_idx`.
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for _, binding in inputs.items():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.dag_acircle_check(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ owner_indx = self.get_owner_indx()
+ bind_owner_indx = binding.get_owner_indx()
+ if owner_indx == bind_owner_indx:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if owner_indx != 0 and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if owner_indx != 0 and bind_owner_indx != 0 and binding.io_type == "output":
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if owner_indx != 0 and bind_owner_indx == 0 and binding.io_type == "input":
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if owner_indx == 0 and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if owner_indx == 0 and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(f"Illegal type:binding type is not same!")
+
+ binding.parents.append(self)
+ # Do acircle check after add the in-degree.
+ if not self.dag_acircle_check(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None, index=0):
+ """init class"""
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.set_index_name(index)
+ self.mod = mod
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+ self.target_host_ = None
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_index_name(self, index):
+ """generate name by index and storage index value"""
+ self.index = index
+ self.name = "mod{}".format(str(index))
+
+ def is_root_mod(self):
+ """use by DAG topology sort, identify if this item is root item and in-degree is 0"""
+ for _, binding in self.input_bindings.bindings.items():
+ if binding.parents:
+ return False
+
+ return True
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for _, binding in self.output_bindings.bindings.items():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def set_target_host(self, host):
+ """set target host that use by build function"""
+ self.target_host_ = host
+
+ def set_build_func(self, build_func):
+ """set build funciton that use by build function"""
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ """set params that use by build function"""
+ self.params_ = params
+
+ def set_target(self, target):
+ """set target that use by build function"""
+ self.target_ = target
+
+ def set_dev(self, dev):
+ """set dev that use by build function"""
+ self.dev_ = dev
+
+ class BindingList:
+ """Use to storage Binding list.
+ Parameters
+ ----------
+
+ owner : ModuleWrapper/PipelineConfig
+ who own this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Illegal name: {self.binding_type}:{key} cannot find")
+
+ self.bindings[key] = PipelineConfig.ModuleWrapper.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ def __init__(self):
+ self.last_mod_indx = 0
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # Sort moudles
+ self.dag_topology_sort()
+
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for _, interface in self.mod_wrapper[mod].output_bindings.bindings.items():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += " " + dep_mname + "." + dep_dname
+ connections_dump += " |" + iname + "\n"
+ else:
+ output[dep_dname] = mname + ".output(" + dname + ")"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += " |output(" + name + ") : " + output[name] + "\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ # self.last_mod_indx start from 1 and be initialize value,
+ # the final value for mod index would get generate by function
+ # dag_topology_sort
Review comment:
In this case can we just let index unset until topological sort? It's weird to see module index has two meanings (adding order and topological order).
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for _, binding in inputs.items():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.dag_acircle_check(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ owner_indx = self.get_owner_indx()
+ bind_owner_indx = binding.get_owner_indx()
+ if owner_indx == bind_owner_indx:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if owner_indx != 0 and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if owner_indx != 0 and bind_owner_indx != 0 and binding.io_type == "output":
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if owner_indx != 0 and bind_owner_indx == 0 and binding.io_type == "input":
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if owner_indx == 0 and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if owner_indx == 0 and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(f"Illegal type:binding type is not same!")
Review comment:
```suggestion
raise RuntimeError(f"Illegal type (%s vs. %s): binding type is not same!" % (self.data_type, binding.data_type))
```
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for _, binding in inputs.items():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.dag_acircle_check(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ owner_indx = self.get_owner_indx()
+ bind_owner_indx = binding.get_owner_indx()
+ if owner_indx == bind_owner_indx:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if owner_indx != 0 and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if owner_indx != 0 and bind_owner_indx != 0 and binding.io_type == "output":
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if owner_indx != 0 and bind_owner_indx == 0 and binding.io_type == "input":
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if owner_indx == 0 and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if owner_indx == 0 and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(f"Illegal type:binding type is not same!")
+
+ binding.parents.append(self)
+ # Do acircle check after add the in-degree.
+ if not self.dag_acircle_check(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None, index=0):
+ """init class"""
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.set_index_name(index)
+ self.mod = mod
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+ self.target_host_ = None
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_index_name(self, index):
+ """generate name by index and storage index value"""
+ self.index = index
+ self.name = "mod{}".format(str(index))
+
+ def is_root_mod(self):
+ """use by DAG topology sort, identify if this item is root item and in-degree is 0"""
+ for _, binding in self.input_bindings.bindings.items():
+ if binding.parents:
+ return False
+
+ return True
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for _, binding in self.output_bindings.bindings.items():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def set_target_host(self, host):
+ """set target host that use by build function"""
+ self.target_host_ = host
+
+ def set_build_func(self, build_func):
+ """set build funciton that use by build function"""
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ """set params that use by build function"""
+ self.params_ = params
+
+ def set_target(self, target):
+ """set target that use by build function"""
+ self.target_ = target
+
+ def set_dev(self, dev):
+ """set dev that use by build function"""
+ self.dev_ = dev
Review comment:
As I mentioned, Python doesn't have private member, so you can just name and access them normally. For example, `mod.set_target_host(host)` -> `mod.target_host = host`.
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,276 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ pipe_config1 = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def pipeline(target):
+ """
+ #Get 3 pipeline module.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+
+ # Prepare batch data for pipeline feeding
+ datas = []
+ for i in range(5):
+ datas.append(np.full(dshape, 3 + i).astype("float32"))
+
+ # Runtime error check
+ pipe_config_check(mod1, mod2, mod3)
+
+ pipe_config = pipeline_executor.PipelineConfig()
+
+ # Create pipeline compute input/output and subgraph dependent relation.
+
+ # Test in key mode for binding find.
+ # pipeline compute input "data_0" would get forward to mod1 as input "data_0"
+ pipe_config["input"]["data_0"].connect(pipe_config[mod1]["input"]["data_0"])
+
+ # pipeline compute input "data_1" would get forward to mod2 as input "data_1"
+ pipe_config["input"]["data_1"].connect(pipe_config[mod2]["input"]["data_1"])
+
+ # mod1 output(0) would get forward to mod2 as input "data_0"
+ pipe_config[mod1]["output"][0].connect(pipe_config[mod2]["input"]["data_0"])
+
+ # mod1 output(1) would get forward to mod3 as input "data_0"
+ pipe_config[mod1]["output"][1].connect(pipe_config[mod3]["input"]["data_0"])
+
+ # mod2 output(0) would get forward to mod3 as input "data_1"
+ pipe_config[mod2]["output"][0].connect(pipe_config[mod3]["input"]["data_1"])
+
+ # mod1 output(2) would get forward as final pipeline compute output(1)
+ pipe_config[mod1]["output"][2].connect(pipe_config["output"]["0"])
+
+ # mod3 output(0) would get forward as final pipeline compute output(2)
+ pipe_config[mod3]["output"][0].connect(pipe_config["output"]["1"])
+ """
+ # print configueration (print(pipe_config)), the expect result like following.
+ #
+ #Inputs
+ # |data_0: mod1:data_0
+ # |data_1: mod2:data_1
+ #
+ #output
+ # |output(1) : mod1.output(2)
+ # |output(2) : mod3.output(0)
+ #
+ #connections
+ # |mod1.output(0)-> mod2.data_0
+ # |mod1.output(1)-> mod3.data_0
+ # |mod2.output(0)-> mod3.data_1
+ """
+
+ """
+ # set other parameter.
+ """
+ pipe_config[mod1].set_target(target[0])
+ pipe_config[mod1].set_dev(target[1])
+
+ pipe_config[mod2].set_target("llvm")
+ pipe_config[mod2].set_dev(tvm.cpu(0))
+
+ pipe_config[mod3].set_target("llvm")
+ pipe_config[mod3].set_dev(tvm.cpu(0))
+
+ """
+ # check if the configuration match expectation.
+ """
+ assert pipe_config.get_config() == get_manual_conf([mod1, mod2, mod3], target)
+
+ """
+ # Test build and create pipeline module
+ """
+ with relay.build_config(opt_level=3):
+ pipeline_mod_config = pipeline_executor.build(pipe_config)
+
+ pipeline_module = pipeline_executor.create(pipeline_mod_config)
+ assert pipeline_module
+
+
+def pipe_config_check(mod1, mod2, mod3):
+ """
+ # try invalid input/output name exepect runtime error
+ """
+ pipe_error = pipeline_executor.PipelineConfig()
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["output"][9]
+
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["input"]["data_9"]
+
+ """
+ # try cirle connection , expect runtime error
+ """
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["output"][0].connect(pipe_error[mod2]["input"]["data_0"])
+ pipe_error[mod2]["output"][0].connect(pipe_error[mod1]["input"]["data_0"])
+
+ """
+ # try wrong module order connection check, expect runtime error.
+ """
+
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["output"][0].connect(pipe_error[mod1]["input"]["data_0"])
+
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["input"]["data_0"].connect(pipe_error[mod1]["input"]["data_0"])
+
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["input"]["data_0"].connect(pipe_error[mod2]["input"]["data_0"])
+
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["output"][0].connect(pipe_error["input"]["data_0"])
+
+ with pytest.raises(RuntimeError):
+ pipe_error["input"]["data_0"].connect(pipe_error[mod1]["output"][0])
+
+ with pytest.raises(RuntimeError):
+ pipe_error["output"]["0"].connect(pipe_error[mod1]["output"][0])
+
+
+def test_pipeline():
+ if pipeline_executor.pipeline_executor_enabled():
+ target_list = tvm.testing.enabled_targets()
+ for target in target_list:
+ pipeline(target)
+
+
+if __name__ == "__main__":
+ test_pipeline()
Review comment:
```suggestion
if __name__ == "__main__":
pytest.main([__file__])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706562006
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
Review comment:
represents
informations -> information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707836660
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
Review comment:
PipelineExecutorFactoryModule not only do current logic as '__init__' function look like, after this initial patch we need to add more function for example "export_library" that to export library into disk just like GraphExecutorFactoryModule did,
hence we may need keep it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707200915
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """Get list of module that represent a subgraph."""
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """The first model has three output."""
+
+ net1_output1 = relay.add(data, mv1)
+ net1_output2 = relay.subtract(data, mv2)
+ net1_output3 = relay.multiply(data, mv3)
+
+ """The second model use first model net1_output1 as first input,
+ here data_net1_output_1 represent net_output1, the second input
+ of this model is data21.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2_output = relay.add(net2, mv3)
+
+ """The third model use the second model net2_output as first input
+ and use the first model net1_output2 as second input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net1_output1, net1_output2, net1_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2_output)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """This function is used to generate manual pipeline configuration."""
+ mod_config = {}
+ """The third output is the final output, the second output is for mod3, the first is for mod2
+ input.
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """This function is used to trigger runtime error by appling wrong logic connection."""
+
+ """Get three pipeline modules here.
+ """
+ (mod1, mod2, mod3), dshape = get_mannual_mod()
+ """The input/output name is illegal and expects a runtime error.
+ """
+ pipe_error = pipeline_executor.PipelineConfig()
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["output"][9]
+
+ with pytest.raises(RuntimeError):
+ pipe_error[mod1]["input"]["data_9"]
+
+ """The connection will cause a circle in DAG and exepects runtime error.
Review comment:
cycle
expects
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707199899
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """Get list of module that represent a subgraph."""
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """The first model has three output."""
+
+ net1_output1 = relay.add(data, mv1)
+ net1_output2 = relay.subtract(data, mv2)
+ net1_output3 = relay.multiply(data, mv3)
+
+ """The second model use first model net1_output1 as first input,
+ here data_net1_output_1 represent net_output1, the second input
+ of this model is data21.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2_output = relay.add(net2, mv3)
+
+ """The third model use the second model net2_output as first input
+ and use the first model net1_output2 as second input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net1_output1, net1_output2, net1_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2_output)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """This function is used to generate manual pipeline configuration."""
+ mod_config = {}
+ """The third output is the final output, the second output is for mod3, the first is for mod2
+ input.
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """This function is used to trigger runtime error by appling wrong logic connection."""
Review comment:
applying
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-896241375
@jroesch , thanks for the follow up, here is the RFC https://github.com/apache/tvm-rfcs/pull/14,
the discussion thread: https://discuss.tvm.apache.org/t/rfc-compute-graph-pipeline-with-new-subgraph-executor/9839/9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-919956408
thanks @huajsj @comaniac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704810868
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline runtime.
+ *
+ * This runtime can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \return The type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the graph executor with graph and context.
+ * \param graph_json The execution graph.
+ * \param module The module containing the compiled functions for the host
+ * processor.
+ * \param ctxs The context of the host and devices where graph nodes will be
+ * executed on.
+ * \param lookup_linked_param_func If given, a PackedFunc invoked to lookup linked parameters
+ * by storage_id. If not given, linked parameters are looked-up using an internal implementation,
+ * which is not compatible with RPCModules.
+ */
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704829306
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,534 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipelineExecutorFactoryModule
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory to storage IRModule list and pipeline configuration.
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module idx and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """return idx if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """check if this interface is global"""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ def __init__(self, mod=None):
+ """init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """generate name by idx and storage idx value"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """use by dag topology sort, identify if this item is root item"""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """use by DAG topology sort, by remove self from binding to reduce child in-degree"""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ class BindingList:
+ """Use to storage Binding list.
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ who own this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.ModuleWrapper.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
+ # topology sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += inf.__repr__() + "\n"
+
+ # get connections
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration in dictionary format."""
+
+ # topology sort to get correct module order in list.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # get pipeline configure
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # build module configuration with pipeline and other parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Do topology sort to get pipeline module order."""
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704884443
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
Review comment:
what are owner name and self name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698900151
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
Review comment:
This is confusing. If users provide a module list then they would expect the first module is named mod0. However, again it would be better if we don't need to rely on index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-897947498
@comaniac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704892002
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,273 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # Get list of module that represent a subgraph.
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # Net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # Net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # Net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
Review comment:
use -> is used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706705453
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
Review comment:
pipeline executor have 2 types input/output, first is global interface that caller use these interface to set input and get output, second is interface of internal subgraph module that build a pipeline from top to down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r701527999
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,276 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """
+ # get list of module that represent a subgraph
+ """
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """
+ # net1 have three output, output3 is final output.
+ """
+
+ net_output1 = relay.add(data, mv1)
+ net_output2 = relay.subtract(data, mv2)
+ net_output3 = relay.multiply(data, mv3)
+
+ """
+ # net2 use net1 output1 as input.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2 = relay.add(net2, mv3)
+
+ """
+ # net3 use net2 output1 and net1 outpu2 as input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net_output1, net_output2, net_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """
+ # This function use to generate manual pipe line configueration,
+ # the result use to verify if the pipe configuration can generate
+ # correct result.
+ """
+ mod_config = {}
+ """
+ # set configure
+ """
+
+ """
+ # third output is final output, second output for mod3, first for mod2
+ # input
+ """
+ pipe_config1 = {
+ "mod_indx": 1,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 2, "input_name": "data_0"}]},
+ {"output_indx": 1, "dependent": [{"mod_indx": 3, "input_name": "data_0"}]},
+ {"output_indx": 2, "dependent": [{"mod_indx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_indx": 2,
+ "output": [
+ {"output_indx": 0, "dependent": [{"mod_indx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_indx": 3,
+ "output": [{"output_indx": 0, "dependent": [{"mod_indx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def pipeline(target):
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707199655
##########
File path: tests/python/relay/test_pipeline_executor.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pytest
+import numpy as np
+import tvm
+import tvm.testing
+from tvm import relay
+from tvm.relay import transform
+from tvm.contrib import graph_executor, pipeline_executor
+
+
+def get_mannual_mod():
+ """Get list of module that represent a subgraph."""
+ mods = []
+ dshape = (3, 3)
+ data = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data21 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net1_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ data_net1_output_2 = relay.var("data_1", relay.TensorType(dshape, "float32"))
+ data_net2_output_1 = relay.var("data_0", relay.TensorType(dshape, "float32"))
+ mvalue1 = np.full((1), 1).astype("float32")
+ mvalue2 = np.full((1), 2).astype("float32")
+ mvalue3 = np.full((1), 3).astype("float32")
+ mv1 = relay.Constant(tvm.nd.array(mvalue1))
+ mv2 = relay.Constant(tvm.nd.array(mvalue2))
+ mv3 = relay.Constant(tvm.nd.array(mvalue3))
+
+ """The first model has three output."""
+
+ net1_output1 = relay.add(data, mv1)
+ net1_output2 = relay.subtract(data, mv2)
+ net1_output3 = relay.multiply(data, mv3)
+
+ """The second model use first model net1_output1 as first input,
+ here data_net1_output_1 represent net_output1, the second input
+ of this model is data21.
+ """
+ net2 = relay.add(data_net1_output_1, mv2)
+ net2 = relay.add(net2, data21)
+ net2_output = relay.add(net2, mv3)
+
+ """The third model use the second model net2_output as first input
+ and use the first model net1_output2 as second input.
+ """
+ net3 = relay.multiply(data_net2_output_1, mv3)
+ net3 = relay.add(net3, data_net1_output_2)
+
+ mods.append(
+ tvm.IRModule.from_expr(
+ relay.Function([data], relay.Tuple([net1_output1, net1_output2, net1_output3]))
+ )
+ )
+ mods.append(tvm.IRModule.from_expr(relay.Function([data_net1_output_1, data21], net2_output)))
+ mods.append(
+ tvm.IRModule.from_expr(relay.Function([data_net1_output_2, data_net2_output_1], net3))
+ )
+
+ return mods, dshape
+
+
+def get_manual_conf(mods, target):
+ """This function is used to generate manual pipeline configuration."""
+ mod_config = {}
+ """The third output is the final output, the second output is for mod3, the first is for mod2
+ input.
+ """
+ pipe_config1 = {
+ "mod_idx": 1,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 2, "input_name": "data_0"}]},
+ {"output_idx": 1, "dependent": [{"mod_idx": 3, "input_name": "data_0"}]},
+ {"output_idx": 2, "dependent": [{"mod_idx": 0, "input_name": "0"}]},
+ ],
+ }
+ mod_config[mods[0]] = {
+ "pipeline": pipe_config1,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": target[0],
+ "dev": target[1],
+ }
+
+ pipe_config2 = {
+ "mod_idx": 2,
+ "output": [
+ {"output_idx": 0, "dependent": [{"mod_idx": 3, "input_name": "data_1"}]},
+ ],
+ }
+ mod_config[mods[1]] = {
+ "pipeline": pipe_config2,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+
+ pipe_config3 = {
+ "mod_idx": 3,
+ "output": [{"output_idx": 0, "dependent": [{"mod_idx": 0, "input_name": "1"}]}],
+ }
+ mod_config[mods[2]] = {
+ "pipeline": pipe_config3,
+ "target_host": None,
+ "mod_name": "default",
+ "build": None,
+ "params": None,
+ "target": "llvm",
+ "dev": tvm.cpu(0),
+ }
+ return mod_config
+
+
+def test_pipe_config_check():
+ """This function is used to trigger runtime error by appling wrong logic connection."""
+
+ """Get three pipeline modules here.
Review comment:
For one line comment, just use #, not """. There are many of them in this function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707592478
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Check whether the binding settings is correct or not.
+ correct connection are following
+ 1. global input to module input
+ 2. module output to global output
+ 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a cycle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building information.
+ """
+
+ def __init__(self, mod=None):
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, interface_type):
+ """Get module interface data type."""
+ if interface_type == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if interface_type == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node, this function used by DAG topological sort."""
Review comment:
changed into "Check whether it is root node and is used by DAG topological sort."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707221173
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Check whether the binding settings is correct or not.
+ correct connection are following
+ 1. global input to module input
+ 2. module output to global output
+ 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a cycle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building information.
+ """
+
+ def __init__(self, mod=None):
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, interface_type):
+ """Get module interface data type."""
+ if interface_type == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if interface_type == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
Review comment:
Set the index value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704883268
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
Review comment:
text format configuration -> configuration as a json string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704891097
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor take Module list and Dependency relations of Modules as
+ * inputs, then execute modules on heterogeneous hardware target in
+ * pipeline parallism mode to improve processing throughput.
+ *
+ * This executor can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \return The type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the pipeline executor with module Array and json text.
+ * \param modules The module list that use for pipeline build.
+ * \param pipeline_json The configuration for module dependent in pipeline.
Review comment:
What is "module dependent in pipeline"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r701521784
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,559 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+ -------
+
+ Returns
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Binding information.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ There are 2 types Binding Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_indx(self):
+ """return index if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.index
+
+ # if not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def dag_acircle_check(self, start, inputs):
+ """check if the DAG that current binding stay is acircle"""
+ for _, binding in inputs.items():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.dag_acircle_check(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ owner_indx = self.get_owner_indx()
+ bind_owner_indx = binding.get_owner_indx()
+ if owner_indx == bind_owner_indx:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if owner_indx != 0 and self.io_type == "input":
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708863450
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
+ to create a pipeline executor.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Build Configuration information.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create a pipeline configuration.
+ string_config[mod_idx] = mconf
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """Pipeline configuration information, this class contains the DAG that expresses
+ the dependency of each module involved by pipeline and the specific parameters
+ of each module build.
+ """
+
+ class Binding:
+ """This class defines the module connections information.
+ The type can only be "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The I/O type of this interface. It can only be "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is an integer such as 0.
+
+ data_type: TensorType
+ The data type of this interface.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child interfaces that depend on this interface.
+ self.bindings = []
+ # Parents interfaces that this interface depend on.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ # Return name of this interface and the name of owner who owns this interface.
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ # If the owner is ModuleWrapper return the owner index, if not return 0.
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use a pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the results computed by the pipeline executor to a caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ # Get all binding information.
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """This is to check whether the DAG containing these input interfaces is acyclic.
+ Parameters
+ ----------
+ start: ModuleWrapper
+ The starting node of the cycle check algorithm.
+
+ inputs: Binding
+ These interfaces are used to connect to each other to build DAG.
+
+ Return
+ ------
+ Return true if there is no cycle in the DAG.
+ """
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Connect the current interface to the destination interface.
+ correct connections as following 1. global input connect to module input,
+ 2. module output connect to global output, 3. module output connect to module input
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707591077
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who owns this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the pipeline executor compute results to caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contains the inputs interfaces is acyclic."""
Review comment:
changed into "This is to check whether the DAG containing these input interfaces is acyclic."
here may need to keep 'interfaces' for that there are multiple input interfaces get involved by checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707827054
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
Review comment:
yes this actually used, here is the use case https://github.com/apache/tvm/pull/8702/files#diff-41c07bdbfe4a927b955fa274635921d878aa75e2b0b2c9e27bafd07d88250ef2R241.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r699027050
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706705201
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
+ return self.output_bindings[idx]
+
+
+class PipelineExecutorFactoryModule(object):
+ """This is a wrapper class which contains GraphExecutorFactoryModule list.
+ and Module configurations.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule.
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698861647
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
Review comment:
IIUC, we can use arbitrary name for graph executor and it's not necessary to be "default". Could you check that? If that's not the case then we probably need to discuss with other folks cuz this doesn't make any sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-912103036
@comaniac , all review comments addressed, please review again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706560706
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor class use module list and dependency relations of modules as
+ * the parameters and executes these modules on heterogeneous pipeline parallel
+ * to improve throughput.
+ *
+ * This executor can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \Return the type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the pipeline executor with module array and json text.
+ * \param modules The module list that used for building pipeline.
+ * \param pipeline_json The configuration of modules dependencies.
+ */
+ void Init(const Array<tvm::runtime::Module>& modules, const std::string& pipeline_json);
+ /*!
+ * \brief Give frontends an access to packed functions.
+ * \param name The name of the function.
+ * \param sptr_to_self The pointer to the module node.
+ * \return The corresponding packed functions.
Review comment:
function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707197383
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline executor.
+ * This executor class use module list and dependency relations of modules as
Review comment:
not fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708797131
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
+ to create a pipeline executor.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Build Configuration information.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create a pipeline configuration.
+ string_config[mod_idx] = mconf
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """Pipeline configuration information, this class contains the DAG that expresses
+ the dependency of each module involved by pipeline and the specific parameters
+ of each module build.
+ """
+
+ class Binding:
+ """This class defines the module connections information.
+ The type can only be "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The I/O type of this interface. It can only be "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is an integer such as 0.
+
+ data_type: TensorType
+ The data type of this interface.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child interfaces that depend on this interface.
+ self.bindings = []
+ # Parents interfaces that this interface depend on.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ # Return name of this interface and the name of owner who owns this interface.
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ # If the owner is ModuleWrapper return the owner index, if not return 0.
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use a pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the results computed by the pipeline executor to a caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ # Get all binding information.
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """This is to check whether the DAG containing these input interfaces is acyclic.
+ Parameters
+ ----------
+ start: ModuleWrapper
+ The starting node of the cycle check algorithm.
+
+ inputs: Binding
+ These interfaces are used to connect to each other to build DAG.
+
+ Return
+ ------
+ Return true if there is no cycle in the DAG.
+ """
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Connect the current interface to the destination interface.
+ correct connections as following 1. global input connect to module input,
Review comment:
Correct connections are as follows:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704902379
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r698852647
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,395 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """check if pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """build module list that can use for pipeline execution.
+
+ Parameters
+ ----------
+ mod_n_configs: Dict[IRModule, Dict[str, Any]]
+ build configuration informaton, structure like following.
+ {IRModule: {"target":target,
+ "target_host":target_host,
+ "params":params,
+ "mod_name"mod_name,
+ "build":build}}
+
+ Returns
+ -------
+ ret: List[IRModule]
+ list of IRModule
+ string_config: Dict[int, Dict[str, any]]
+ pipeline configuration
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ #for _, (ir_mod, mod_config) in enumerate(mod_n_configs.items()):
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_indx = mconf["mod_indx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_indx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ # return PipeModuleConfig
+ return PipeModuleConfig(mods, string_config)
+
+
+def create(pipe_mod_config):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+
+ pipe_mod_config : PipeModuleConfig
+ class to storage IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_mod_config)
+
+class PipelineModule(object):
+ """Wrapper runtime module. This is a thin wrapper of the underlying TVM module.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphModule]
+ The internal tvm module that holds the actual graph functions.
+ pipeline_config : Dict[IRModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods_ = pipe_mod_config.pipeline_mods_
+ self.mod_config_ = pipe_mod_config.mods_config_
+ mods, config = self.graph_executor_create(self.pipeline_mods_, self.mod_config_)
+ assert pipeline_executor_enabled(), \
+ "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipelinecreate = tvm._ffi.get_global_func("tvm.pipeline_executor.create",
+ allow_missing=False)
+ assert pipelinecreate
+ module = pipelinecreate(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return string format config.
+
+ Parameters
+ ----------
+
+ pipeline_mods : List[IRModule]
+ list of IRModule
+
+ mod_config : Dict[int, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[GraphModule]
+ Runtime graph module.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """Pipeline Configuration Class, in this class there are 2 internal class,
+ first is Module which use to represent Module, second is Interface which use
+ to represent Module input/output and Pipeline Module input/output, by setting
+ dependency relation between Interfaces this class can build the module
+ connection relation.
+
+ The class Hierarchical as following.
+ PipelineConfig ---> ModuleWrapper ---> Interface(input/output)
+ """
+
+ class ModuleWrapper:
+ """The class use use to represent Module and storage module index and
+ Interface information.
+ """
+
+ class Interface:
+ """The class that use to storage module connection information.
+ There are 2 types Interface Input:1 Output:2
+ Parameters
+ ----------
+
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like index, module name
+
+ itype : integer
+ Interface type, 1 is input interface, 2 is output interface
+
+ name : str/integer
+ Interface name, for input that is string for example "data0"
+ for output that is integer for example 0.
+ """
+
+ def __init__(self, owner, itype, name):
+ self.owner_ = owner
+ self.itype_ = itype
+ self.name_ = str(name)
+ self.dependent_ = []
+
+ def get_name(self):
+ mname = ""
+ if self.owner_:
+ mname = self.owner_.name_
+
+ return mname, self.name_
+
+ def get_owner_indx(self):
+ return self.owner_.indx_
+
+ def get_dependent_str(self):
+ name = ""
+ for dependent in self.dependent_:
+ mname, dname = dependent.get_name()
+ name = name + (mname + ":output(" + dname if self.itype_ == 2 else "")
+ name = name + (")" if self.itype_ == 2 else mname + ":" + dname)
+ return name
+
+ def add_dependent(self, dependent):
+ """
+ # check if the dependency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to next module input
+ # 3. module output to global output
+ """
+ owner_indx = self.get_owner_indx()
+ dep_owner_indx = dependent.get_owner_indx()
+ assert owner_indx != dep_owner_indx, f"can not set self as dependent."
+ assert not (
+ owner_indx > dep_owner_indx
+ and not (dependent.itype_ == 2 and dep_owner_indx == 0)
+ ), f"dependent only can be next module interface or global output."
+ assert not (
+ owner_indx == 0 and dependent.itype_ != 1
+ ), f"global input only can set dependent with module input."
+
+ self.dependent_.append(dependent)
+
+ def __init__(self, indx=0):
+ self.indx_ = indx
+ self.name_ = "mod" + str(indx) if indx else ""
+ self.interfaces_ = {1: {}, 2: {}}
+ self.target_host_ = None
+ self.mod_name_ = "default"
+ self.build_func_ = None
+ self.params_ = None
+ self.target_ = None
+ self.dev_ = None
+
+
+ def get_interface(self, itype, name):
+ if name not in self.interfaces_[itype]:
+ self.interfaces_[itype][name] = self.Interface(self, itype, name)
+
+ return self.interfaces_[itype][name]
+
+ def input(self, name):
+ return self.get_interface(1, name)
+
+ def output(self, index):
+ return self.get_interface(2, index)
+
+ def set_target_host(self, host):
+ self.target_host_ = host
+
+ def set_mod_name(self, name):
+ self.mod_name_ = name
+
+ def set_build_func(self, build_func):
+ self.build_func_ = build_func
+
+ def set_params(self, params):
+ self.params_ = params
+
+ def set_target(self, target):
+ self.target_ = target
+
+ def set_dev(self, dev):
+ self.dev_ = dev
+
+ def __init__(self, mods):
+ self.pipe_module_name_ = "pipeline_module"
+ self.mod_wrapper = {m: self.ModuleWrapper(i + 1) for m, i in zip(mods, range(len(mods)))}
+ self.mod_wrapper[self.pipe_module_name_] = self.ModuleWrapper(0)
Review comment:
yes, this more clear, I would apply the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704946459
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
Review comment:
changed into "Here to check if the DAG that build by bindings connections is acyclic."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r701632127
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <memory>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "../file_utils.h"
Review comment:
Remove unused headers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707834124
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
Review comment:
Here may need to keep PipelineModule for exporting function like setinput, getoutput etc that not included in this initial patch.
removed create function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#issuecomment-918874900
@masahi all review comments addressed, please review again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708835908
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
+ to create a pipeline executor.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Build Configuration information.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create a pipeline configuration.
+ string_config[mod_idx] = mconf
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """Pipeline configuration information, this class contains the DAG that expresses
+ the dependency of each module involved by pipeline and the specific parameters
+ of each module build.
+ """
+
+ class Binding:
+ """This class defines the module connections information.
+ The type can only be "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The I/O type of this interface. It can only be "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is an integer such as 0.
+
+ data_type: TensorType
+ The data type of this interface.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child interfaces that depend on this interface.
+ self.bindings = []
+ # Parents interfaces that this interface depend on.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ # Return name of this interface and the name of owner who owns this interface.
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ # If the owner is ModuleWrapper return the owner index, if not return 0.
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use a pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the results computed by the pipeline executor to a caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ # Get all binding information.
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """This is to check whether the DAG containing these input interfaces is acyclic.
+ Parameters
+ ----------
+ start: ModuleWrapper
+ The starting node of the cycle check algorithm.
+
+ inputs: Binding
+ These interfaces are used to connect to each other to build DAG.
+
+ Return
+ ------
+ Return true if there is no cycle in the DAG.
+ """
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Connect the current interface to the destination interface.
+ correct connections as following 1. global input connect to module input,
+ 2. module output connect to global output, 3. module output connect to module input
+
+ Parameters
+ ----------
+ binding: Binding
+ The destination of this connection.
+ """
+
+ # Check whether the binding setting is correct or not.
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # Check whether the data types of the source and destination are the same.
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
Review comment:
Fixed.
the in-degree count by the length of binding.parents, changed the comments into
"Do acyclic check after increasing the in-degree of child node by setting current interface as a parent of the child node"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707836660
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,525 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return module list and module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
Review comment:
PipelineExecutorFactoryModule not only do current logic as '__init__' function look like, after this initial patch we need to add more function for example "export_library" that to export library into disk just like GraphExecutorFactoryModule did, hence we may need keep it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704735393
##########
File path: src/runtime/pipeline/pipeline_executor.h
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \brief pipeline executor
+ * \file pipeline_executor.h
+ */
+#ifndef TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#define TVM_RUNTIME_PIPELINE_PIPELINE_EXECUTOR_H_
+#include <tvm/runtime/registry.h>
+
+#include <string>
+namespace tvm {
+namespace runtime {
+/*!
+ * \brief pipeline runtime.
+ *
+ * This runtime can be acccesibly in various language via
+ * TVM runtime PackedFunc API.
+ */
+class TVM_DLL PipelineRuntime : public ModuleNode {
+ public:
+ /*!
+ * \return The type key of the executor.
+ */
+ const char* type_key() const final { return "PipelineRuntime"; }
+ /*!
+ * \brief Initialize the graph executor with graph and context.
+ * \param graph_json The execution graph.
+ * \param module The module containing the compiled functions for the host
+ * processor.
+ * \param ctxs The context of the host and devices where graph nodes will be
+ * executed on.
+ * \param lookup_linked_param_func If given, a PackedFunc invoked to lookup linked parameters
+ * by storage_id. If not given, linked parameters are looked-up using an internal implementation,
+ * which is not compatible with RPCModules.
+ */
+ void Init(const Array<tvm::runtime::Module>& modules, const std::string& pipeline_json);
+ /*!
+ * \brief Get member function to front-end.
Review comment:
Give frontends the access to packed functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704889010
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"can not set self as binding.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module only can start binding from output!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Module output can not binding with module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Module output can not binding with global input!")
+
+ if self.is_global_interface() and self.io_type != "input":
+ raise RuntimeError(f"Global only can start binding from input!")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input only can set binding with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list, it can be ModuleWrapper or PipelineConfig
+
+ type_name : str
+ The type of this binding list. It can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ """return binding data type"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ """return item by key"""
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Cannot find {key} in binding list {self.binding_type}")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """Module Wrapper with information like module index, binding information
+ and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ """Check if self equl other"""
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ """Get item by key"""
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get module input/output data type."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Set index and generating name by index"""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Identify if this item is root item, used by DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove self from binding to reduce child in-degree, used by DAG topological sort."""
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration in string type"""
Review comment:
in string type -> as a string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r704904480
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,533 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if pipeline executor get enabled.
+
+ Return
+ -------
+ enable: bool
+ Return pipeline executor is enabled or not.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build IRModule with configurations of pipe_configs then
+ return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ build configuration informaton.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ the class that wrap module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device config
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # if there is a self defined build function then use it.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # build IRModule
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration
+ string_config[mod_idx] = mconf
+ # associate mod with device
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ Executor factory with IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ modules and modules dependency configuration informaiton.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return text format configuration.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ list of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ modules dependency configuration informaiton.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ mods configuration
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """The class that use to storage module connection information.
+ The binding can be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class that own this interface, in such class there are
+ Module information like idx, module name
+
+ io_type : str
+ The type of this binding. It can be either "input" or "output".
+
+ name : str/integer
+ Binding name, for input it is string such as "data0";
+ for output it is the idx integer such as 0.
+ """
+
+ def __init__(self, owner, stype, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = stype
+ self.name = str(name)
+ # These item that have dependency relation with self
+ self.bindings = []
+ # The item that self depend
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Get owner name and self name"""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0"""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """Check if this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data), exepect like |data_0: mod1:data_0"""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """Check if the DAG that current binding stay is acircle"""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check if the bindendency setting correct.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to moudle input
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] huajsj commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
huajsj commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706708846
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r706562440
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
+ binding informations and building informations.
+ """
+
+ def __init__(self, mod=None):
+ """Init class"""
+ self.target_host = None
+ self.build_func = None
+ self.params = None
+ self.target = None
+ self.name = None
+ self.dev = None
+ self.idx = None
+ self.mod = mod
+ self.input_params = InferType()(mod)["main"].params
+ self.output_values = InferType()(mod)["main"].checked_type.ret_type
+ self.input_bindings = PipelineConfig.BindingList(self, "input")
+ self.output_bindings = PipelineConfig.BindingList(self, "output")
+
+ def __eq__(self, other):
+ if isinstance(other, PipelineConfig.ModuleWrapper):
+ return self.mod == other.mod
+
+ return False
+
+ def __getitem__(self, key):
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found!")
+
+ def get_data_type(self, key, stype):
+ """Get the data type of the input or output interface of the module."""
+ if stype == "input":
+ for param in self.input_params:
+ if param.name_hint == key:
+ return param._checked_type_
+
+ if stype == "output":
+ if isinstance(self.output_values, tvm.ir.type.TupleType):
+ if int(key) < len(self.output_values.fields):
+ return self.output_values.fields[int(key)]
+ elif int(key) == 0:
+ return self.output_values
+
+ return None
+
+ def set_idx_name(self, idx):
+ """Sepecify the index value and generate the module name."""
+ self.idx = idx
+ self.name = "mod{}".format(str(idx))
+
+ def is_root_mod(self):
+ """Check whether it is root node and use it in DAG topological sort."""
+ return all([not b.parents for b in self.input_bindings.bindings.values()])
+
+ def remove_self_from_bindings(self):
+ """Remove itself from child reference to reduce child node in-degree
+ and use it in DAG topological sort.
+ """
+ for binding in self.output_bindings.bindings.values():
+ for child in binding.bindings:
+ if binding in child.parents:
+ child.parents.remove(binding)
+
+ def __init__(self):
+ self.mod_wrapper = {}
+ self.input_bindings = self.BindingList(self, "input")
+ self.output_bindings = self.BindingList(self, "output")
+
+ def __str__(self):
+ """ Get configuration as string"""
+ # topological sort to get correct module order in list.
+ self.dag_topology_sort()
+ # get input
+ input_dump = "Inputs\n"
+ for input_name in self.input_bindings.bindings:
+ inf = self.input_bindings.bindings[input_name]
+ input_dump += str(inf) + "\n"
+
+ # get connections.
+ output = {}
+ connections_dump = "\nconnections\n"
+ for mod in self.mod_wrapper:
+ for interface in self.mod_wrapper[mod].output_bindings.bindings.values():
+ if interface.bindings:
+ mname, dname = interface.get_name()
+ iname = mname + ".output(" + dname + ")->"
+ for dep in interface.bindings:
+ dep_mname, dep_dname = dep.get_name()
+ if isinstance(dep.io_owner, PipelineConfig.ModuleWrapper):
+ iname += f" {dep_mname}.{dep_dname}"
+ connections_dump += f" |{iname}\n"
+ else:
+ output[dep_dname] = f"{mname}.output({dname})"
+
+ # get output
+ output_dump = "\noutput\n"
+ for name in sorted(output.keys()):
+ output_dump += f" |output({name}) : {output[name]}\n"
+
+ return input_dump + output_dump + connections_dump
+
+ def __getitem__(self, key):
+ if isinstance(key, tvm.ir.module.IRModule):
+ if key not in self.mod_wrapper:
+ self.mod_wrapper[key] = self.ModuleWrapper(key)
+ return self.mod_wrapper[key]
+
+ if isinstance(key, str):
+ if key == "input":
+ return self.input_bindings
+ if key == "output":
+ return self.output_bindings
+
+ raise RuntimeError(f"{key} not found.")
+
+ def get_config(self):
+ """ Get configuration information in dictionary form."""
+
+ # Use topological sort to get correct order of modules.
+ self.dag_topology_sort()
+ mconfig = {}
+ for mod in self.mod_wrapper:
+ # Generate pipeline configure.
+ mconf = {}
+ output_conf = []
+ module = self.mod_wrapper[mod]
+ for _, binding in module.output_bindings.bindings.items():
+ dep_conf = []
+ output = {}
+ if binding.bindings:
+ for dep in binding.bindings:
+ dep_item = {}
+ _, dname = dep.get_name()
+ dep_item["mod_idx"] = dep.get_owner_idx()
+ dep_item["input_name"] = dname
+ dep_conf.append(dep_item)
+
+ # Ouput_idx start from 0.
+ output["output_idx"] = int(binding.name)
+ output["dependent"] = dep_conf
+ output_conf.append(output)
+
+ mconf["mod_idx"] = module.idx
+ mconf["output"] = output_conf
+
+ # Build configuration with parameters.
+ mconfig[mod] = {
+ "pipeline": mconf,
+ "target_host": module.target_host,
+ "mod_name": "default",
+ "build": module.build_func,
+ "params": module.params,
+ "target": module.target,
+ "dev": module.dev,
+ }
+
+ return mconfig
+
+ def dag_topology_sort(self):
+ """ Use topological sort to get order of pipeline modules."""
+ mlist = []
+ mod_wrapper = self.mod_wrapper.copy()
+ while mod_wrapper:
+ temp_list = []
+ for mod, wrapper in mod_wrapper.items():
+ if wrapper.is_root_mod():
+ temp_list.append(mod)
+ wrapper.remove_self_from_bindings()
+
+ for mod in temp_list:
+ mod_wrapper.pop(mod, None)
+
+ mlist += temp_list
+
+ for mod, i in zip(mlist, range(len(mlist))):
+ self.mod_wrapper[mod].set_idx_name(i + 1)
+
+ def get_mod_idx(self, mod):
+ """Return module index."""
+ idx = self.mod_wrapper[mod].idx
+ return idx
+
+ def pipe_input(self, name):
+ """Return the corresponding input binding interface accordding the name."""
+ return self.input_bindings[name]
+
+ def pipe_output(self, idx):
+ """Return the corresponding output binding interface accordding the name."""
Review comment:
according to
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r707206716
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,529 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Use pipe_config to build and return Module list and Module dependency configuration.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Configuration information for build.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ A class that wraps module list and module dependency configuration.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ # Get mod device configuration.
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ # Build.
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create pipeline configuration.
+ string_config[mod_idx] = mconf
+ # Set device.
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+def create(pipe_executor_factory_module):
+ """Create a pipeline runtime executor.
+
+ Parameters
+ ----------
+ pipe_executor_factory_module : PipelineExecutorFactoryModule
+ It is wrapper class which include IRModule list and pipeline configuration.
+
+ Returns
+ -------
+ submodule : PipelineModule
+ Runtime pipeline module.
+ """
+
+ return PipelineModule(pipe_executor_factory_module)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module.
+
+ Parameters
+ ----------
+ pipeline_config : Dict[GraphExecutorFactoryModule, Dict[str, Any]]
+ Modules and modules dependency configuration informaitons.
+ """
+
+ def __init__(self, pipe_mod_config):
+ self.pipeline_mods = pipe_mod_config.pipeline_mods
+ self.mod_config = pipe_mod_config.mods_config
+ mods, config = self.graph_executor_create(self.pipeline_mods, self.mod_config)
+ assert (
+ pipeline_executor_enabled()
+ ), "Pipeline executor is not enabled. Please \
+ re-build TVM with USE_PIPELINE_EXECUTOR=ON"
+ pipeline_create = tvm._ffi.get_global_func(
+ "tvm.pipeline_executor.create", allow_missing=False
+ )
+ assert pipeline_create
+ module = pipeline_create(mods, config)
+
+ self.module_ = module
+
+ def graph_executor_create(self, pipeline_mods, mod_config):
+ """Create graph_executor list and return configuration as a json string.
+
+ Parameters
+ ----------
+ pipeline_mods : List[GraphExecutorFactoryModule]
+ List of GraphExecutorFactoryModule
+
+ mod_config : Dict[str, Any]
+ Modules dependency configuration information.
+
+ Returns
+ -------
+ mods : List[Module]
+ Module list.
+
+ mod_config : str
+ Mods configuration.
+ """
+
+ mods = []
+ for pipeline_mod in pipeline_mods:
+ mod = graph_executor.GraphModule(
+ pipeline_mod["default"](pipeline_mods[pipeline_mod]["dev"])
+ )
+ mods.append(mod.module)
+
+ return mods, json.dumps(mod_config)
+
+
+class PipelineConfig(object):
+ """The wrapper of each module to be pipelined. The wrapper mainly includes the
+ module itself as well as the binding that represents the connections of this
+ module's inputs and outputs to other modules.
+ """
+
+ class Binding:
+ """This class define the module connection information.
+ The type can only be either "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The type of this interface. It can only be either "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is the
+ idx integer such as 0.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child nodes.
+ self.bindings = []
+ # Parents nodes.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ """Return the interface name and name of owner who own this interface."""
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ """Return owner idex if owner is ModuleWrapper, if not return 0."""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ # If not ModuleWrapper then owner is PipelineConfig, return 0
+ # to identify this is global interface
+ return 0
+
+ def is_global_interface(self):
+ """It is to check whether this interface is global interface."""
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ """Get all binding(input data) informations that looks like '|data_0: mod1:data_0'."""
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """It is to check whether the DAG that contain the inputs interfaces is acyclic."""
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """
+ # Check whether the binding settings is correct or not.
+ # correct connection are following
+ # 1. global input to module input
+ # 2. module output to global output
+ # 3. module output to module input
+ """
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # check if the source and target data_type same
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
+ if not self.check_dag_acyclic(
+ binding.io_owner, self.io_owner.input_bindings.bindings
+ ):
+ raise RuntimeError(f"Illegal connection: Cause a circle!")
+
+ class BindingList:
+ """Container for bindings(input or output interface).
+
+ Parameters
+ ----------
+ owner : ModuleWrapper/PipelineConfig
+ The owner of this list can be ModuleWrapper or PipelineConfig.
+
+ type_name : str
+ The type of this binding list can be either "input" or "output".
+ """
+
+ def __init__(self, owner, type_name):
+ self.bindings = {}
+ self.io_owner = owner
+ self.binding_type = type_name
+
+ def get_binding_data_type(self, key):
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.get_data_type(key, self.binding_type)
+ return None
+
+ def __getitem__(self, key):
+ if key not in self.bindings:
+ data_type = self.get_binding_data_type(key)
+ if not data_type and isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ raise RuntimeError(f"Can not find {key} in binding list {self.binding_type}.")
+
+ self.bindings[key] = PipelineConfig.Binding(
+ self.io_owner, self.binding_type, key, data_type
+ )
+
+ return self.bindings[key]
+
+ class ModuleWrapper:
+ """This class is a wrapper that represent the module, contains module informations,
Review comment:
not fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708797835
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
+ to create a pipeline executor.
+
+ Parameters
+ ----------
+ pipe_configs: PipelineConfig
+ Build Configuration information.
+
+ Returns
+ -------
+ ret: PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+ mods = {}
+ mod_n_configs = pipe_configs.get_config()
+ config_len = len(mod_n_configs)
+ string_config = [{} for _ in range(config_len)]
+ for ir_mod, mod_config in mod_n_configs.items():
+ mconf = mod_config["pipeline"].copy()
+ mod_idx = mconf["mod_idx"] - 1
+ dev = mod_config["dev"]
+ target = mod_config["target"]
+ build_func = relay.build
+ # Check whether there is a customized build function.
+ if "build" in mod_config and mod_config["build"]:
+ build_func = mod_config["build"]
+
+ mod = build_func(
+ ir_mod,
+ target,
+ params=mod_config["params"],
+ target_host=mod_config["target_host"],
+ mod_name=mod_config["mod_name"],
+ )
+
+ mconf["dev"] = "{},{}".format(dev.device_type, dev.device_id)
+ # Create a pipeline configuration.
+ string_config[mod_idx] = mconf
+ mods[mod] = {"dev": dev}
+
+ return PipelineExecutorFactoryModule(mods, string_config)
+
+
+class PipelineModule(object):
+ """Wrapper of runtime module, caller can use this module to set parameters and get outputs.
+
+ Parameters
+ ----------
+ module : PipelineExecutorFactoryModule
+ Common interface for pipeline executor factory modules.
+ """
+
+ def __init__(self, module):
+ self.module = module.module
+
+
+class PipelineConfig(object):
+ """Pipeline configuration information, this class contains the DAG that expresses
+ the dependency of each module involved by pipeline and the specific parameters
+ of each module build.
+ """
+
+ class Binding:
+ """This class defines the module connections information.
+ The type can only be "input" or "output".
+
+ Parameters
+ ----------
+ owner : ModuleWrapper
+ The class who owns this interface.
+
+ io_type : str
+ The I/O type of this interface. It can only be "input" or "output".
+
+ name : str/integer
+ Name, for input it is string such as "data0", for output it is an integer such as 0.
+
+ data_type: TensorType
+ The data type of this interface.
+ """
+
+ def __init__(self, owner, io_type, name, data_type=None):
+ self.io_owner = owner
+ self.io_type = io_type
+ self.name = str(name)
+ # Child interfaces that depend on this interface.
+ self.bindings = []
+ # Parents interfaces that this interface depend on.
+ self.parents = []
+
+ self.data_type = data_type
+
+ def get_name(self):
+ # Return name of this interface and the name of owner who owns this interface.
+ owner_name = ""
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ owner_name = self.io_owner.name
+
+ return owner_name, self.name
+
+ def get_owner_idx(self):
+ # If the owner is ModuleWrapper return the owner index, if not return 0.
+ if isinstance(self.io_owner, PipelineConfig.ModuleWrapper):
+ return self.io_owner.idx
+
+ return 0
+
+ def is_global_interface(self):
+ """The global interface is the interface visible to the caller which use a pipeline
+ executor, the global input interface is responsible for passing parameters to the
+ internal module interface, and the global output interface is responsible for
+ outputting the results computed by the pipeline executor to a caller.
+ """
+ return not isinstance(self.io_owner, PipelineConfig.ModuleWrapper)
+
+ def __repr__(self):
+ # Get all binding information.
+ ret = " |{}: ".format(self.name)
+ for binding in self.bindings:
+ mname, dname = binding.get_name()
+ ret += "{0}:{1} ".format(mname, dname)
+ return ret
+
+ def check_dag_acyclic(self, start, inputs):
+ """This is to check whether the DAG containing these input interfaces is acyclic.
+ Parameters
+ ----------
+ start: ModuleWrapper
+ The starting node of the cycle check algorithm.
+
+ inputs: Binding
+ These interfaces are used to connect to each other to build DAG.
+
+ Return
+ ------
+ Return true if there is no cycle in the DAG.
+ """
+ for binding in inputs.values():
+ if start == binding.io_owner:
+ return False
+ for p in binding.parents:
+ if not self.check_dag_acyclic(start, p.io_owner.input_bindings.bindings):
+ return False
+
+ return True
+
+ def connect(self, binding):
+ """Connect the current interface to the destination interface.
+ correct connections as following 1. global input connect to module input,
+ 2. module output connect to global output, 3. module output connect to module input
+
+ Parameters
+ ----------
+ binding: Binding
+ The destination of this connection.
+ """
+
+ # Check whether the binding setting is correct or not.
+ if self.io_owner == binding.io_owner:
+ raise RuntimeError(f"Can not bind itself.")
+
+ if not self.is_global_interface() and self.io_type == "input":
+ raise RuntimeError(f"Module can only bind from output interface!")
+
+ if (
+ not self.is_global_interface()
+ and not binding.is_global_interface()
+ and binding.io_type == "output"
+ ):
+ raise RuntimeError(f"Can not bind module output with another module output!")
+
+ if (
+ not self.is_global_interface()
+ and binding.is_global_interface()
+ and binding.io_type == "input"
+ ):
+ raise RuntimeError(f"Can not bind module output with global input!")
+
+ if self.is_global_interface() and self.io_type == "output":
+ raise RuntimeError(f"Global output can not be used as binding start point.")
+
+ if self.is_global_interface() and binding.io_type != "input":
+ raise RuntimeError(f"Global input can only bind with module input.")
+
+ self.bindings.append(binding)
+ if not self.is_global_interface():
+ # Check whether the data types of the source and destination are the same.
+ if (
+ isinstance(binding.io_owner, PipelineConfig.ModuleWrapper)
+ and self.data_type != binding.data_type
+ ):
+ raise RuntimeError(
+ f"Illegal type (%s vs. %s): binding type is not same!"
+ % (self.data_type, binding.data_type)
+ )
+
+ binding.parents.append(self)
+ # Do acyclic check after increase the in-degree.
Review comment:
increasing
in-degree means `binding.parents`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #8702: [Runtime] Pipeline Executor Initial patch.
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #8702:
URL: https://github.com/apache/tvm/pull/8702#discussion_r708795215
##########
File path: python/tvm/contrib/pipeline_executor.py
##########
@@ -0,0 +1,539 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Pipeline executor that executes a series of modules in a pipeline fashion."""
+import json
+import tvm._ffi
+from tvm import relay
+from tvm.relay.transform import InferType
+from tvm.contrib import graph_executor
+
+
+def pipeline_executor_enabled():
+ """Check if the pipeline executor is enabled.
+
+ Return
+ -------
+ enable: bool
+ Return whether the pipeline executor is enabled.
+ """
+ return tvm._ffi.get_global_func("tvm.pipeline_executor.create", allow_missing=True) is not None
+
+
+def build(pipe_configs):
+ """Build these modules used in the pipeline executor, then use these modules and configuration
Review comment:
What are `these modules`. Maybe you can just drop "these".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org