You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/09/28 16:21:29 UTC
[GitHub] [incubator-tvm] leandron opened a new pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
leandron opened a new pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578
Introduces the `tvmc run` subcommand on `tvmc`, the aims to enable execution of compiled modules using `tvmc compile`:
* Include options to locally or remotelly using RPC
* Include support to cpu and gpu devices
* Introduces a new function to parse RPC command line strings (will be backported to `tvmc tune` in a future PR)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r497298591
##########
File path: tests/python/driver/tvmc/conftest.py
##########
@@ -110,10 +112,33 @@ def onnx_resnet50():
@pytest.fixture(scope="session")
def tflite_compiled_module_as_tarfile(tmpdir_factory):
+
+ # Not all CI environments will have TFLite installed
+ # so we need to safely skip this fixture that will
+ # crash the tests that rely on it.
+ # As this is a pytest.fixture, we cannot take advantage
+ # of pytest.importorskip. Using the block below instead.
+ try:
+ import tflite
+ except ImportError:
+ print("Cannot import tflite, which is required by tflite_compiled_module_as_tarfile.")
+ return ""
+
target_dir = tmpdir_factory.mktemp("data")
graph, lib, params, _ = get_sample_compiled_module(target_dir)
Review comment:
This refers to `compile`, which is the mechanism we are using here to generate this test fixture. The underlying API here is `relay.build` (not `tvm.build`, as far as I could check).
Here is the relevant snippet:
https://github.com/apache/incubator-tvm/blob/fde8d432acd2d7964607c4f9216720298db04007/python/tvm/driver/tvmc/compiler.py#L192-L203
With regards to the `params` embedded into `lib`, I think this is something we could improve in a future PR? It is likely to have more implications in the tool, that I'd need to investigate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#issuecomment-702149820
something went wrong with CI. Retriggering it now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496728910
##########
File path: python/tvm/driver/tvmc/common.py
##########
@@ -63,3 +72,32 @@ def convert_graph_layout(mod, desired_layout):
raise TVMCException(
"Error converting layout to {0}: {1}".format(desired_layout, str(err))
)
+
+
+def tracker_host_port_from_cli(rpc_tracker_str):
+ """Alter the layout of the input graph.
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498564908
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
+ "making it take longer to be generated.",
+ )
+ parser.add_argument(
+ "--repeat", metavar="N", type=int, default=1, help="repeat the run n times. Defaults to '1'"
+ )
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
Review comment:
thanks for explainment, this is better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496773214
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
Review comment:
Yep, will do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r497300660
##########
File path: tests/python/driver/tvmc/test_runner.py
##########
@@ -0,0 +1,114 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import pytest
+import numpy as np
+
+from tvm.driver import tvmc
+
+
+def test_generate_tensor_data_zeros():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "zeros")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data_ones():
+ expected_shape = (224, 224)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "ones")
+
+ assert sut.shape == (224, 224)
+
+
+def test_generate_tensor_data_random():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "random")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data__type_unknown():
+ with pytest.raises(tvmc.common.TVMCException) as e:
+ tvmc.runner.generate_tensor_data((2, 3), "float32", "whatever")
+
+
+def test_format_times__contains_header():
+ sut = tvmc.runner.format_times([60, 120, 12, 42])
+ assert "std (s)" in sut
+
+
+def test_get_top_results_keep_results():
+ fake_outputs = {"output_0": np.array([[1, 2, 3, 4], [5, 6, 7, 8]])}
+ number_of_results_wanted = 3
+ sut = tvmc.runner.get_top_results(fake_outputs, number_of_results_wanted)
+
+ expected_number_of_lines = 2
+ assert len(sut) == expected_number_of_lines
+
+ expected_number_of_results_per_line = 3
+ assert len(sut[0]) == expected_number_of_results_per_line
+ assert len(sut[1]) == expected_number_of_results_per_line
+
+
+def test_get_top_results_keep_results__limit_bigger_than_returned():
+ fake_outputs = {"output_0": np.array([[1, 2, 3, 4], [5, 6, 7, 8]])}
+ number_of_results_wanted = 6
+ sut = tvmc.runner.get_top_results(fake_outputs, number_of_results_wanted)
+
+ expected_number_of_lines = 2
+ assert len(sut) == expected_number_of_lines
+
+ # despite 'number_of_results_wanted' being 6,
+ # we limit to the maximum available, in this
+ # case it is 4.
Review comment:
This is mainly checking the logic into `get_top_results()`, just to make sure that boundaries are respected. Looking at it again, I don't think this test case is adding much, so if you agree I'll remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498565839
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
Review comment:
IMO this CLI is facing to end-users more than the developers, so I supposed they care the end-to-end time more than just the inference time. However, I agree with you that rephrasing the description to avoid inference time might be more accurate.
OHTH, given that this PR is in the list of 0.7 release, would you mind letting the PR in first and fix this issue in the follow-up PR along with other TODOs, since the CI takes a long time?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496873622
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
+
+ if hostname:
+ # Remote RPC
+ if rpc_key:
+ logger.debug("running on remote RPC tracker with key %s", rpc_key)
+ remote = request_remote(rpc_key, hostname, port, timeout=1000)
+ else:
+ logger.debug("running on remote RPC with no key")
+ remote = rpc.connect(hostname, port)
+ else:
+ # Local
+ logger.debug("running a local session")
+ remote = rpc.LocalSession()
+
+ remote.upload(os.path.join(tmp_dir, "mod.so"))
+ remote_lib = remote.load_module("mod.so")
+
+ # TODO expand to other supported devices, as listed in tvm.rpc.client (@leandron)
+ logger.debug("remote device is %s", device)
+ ctx = remote.cpu() if device == "cpu" else remote.gpu()
+
+ if profile:
+ logger.debug("creating runtime with profiling enabled")
+ module = debug_runtime.create(graph, remote_lib, ctx, dump_root="./prof")
+ else:
+ logger.debug("creating runtime with profiling disabled")
+ module = runtime.create(graph, remote_lib, ctx)
+
+ logger.debug("load params into the runtime module")
+ module.load_params(params)
+
+ shape_dict, dtype_dict = get_input_info(graph, params)
+ inputs_dict = make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode)
+
+ logger.debug("setting inputs to the module")
+ module.set_input(**inputs_dict)
+
+ # Run must be called explicitly if profiling
+ if profile:
+ logger.debug("running the module with profiling enabled")
+ module.run()
+
+ # create the module time evaluator (returns a function)
+ timer = module.module.time_evaluator("run", ctx, 1, repeat=repeat)
+ # call the evaluator function to invoke the module and save execution times
+ prof_result = timer()
+ # collect a list of execution times from the profiling results
+ times = prof_result.results
+
+ logger.debug("collecting the output tensors")
+ num_outputs = module.get_num_outputs()
+ outputs = {}
+ for i in range(num_outputs):
+ output_name = "output_{}".format(i)
+ outputs[output_name] = module.get_output(i).asnumpy()
+
+ return outputs, times
+
+
+def get_top_results(outputs, max_results):
+ """Return the top n results from the output tensor.
+
+ This function is primarily for image classification and will
+ not necessarily generalise.
+
+ Parameters
+ ----------
+ outputs : dict
+ Outputs dictionary - {output_name: np.array}.
+ max_results : int
+ Number of results to return
+
+ Returns
+ -------
+ top_results : np.array
+ Results array of shape (2, n).
+ The first row is the indices and the second is the values.
Review comment:
That was being used mostly to models like MobileNet, ResNet, etc. that would give you a (2, n) output tensor. @mbaret can give more context.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] mbaret commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
mbaret commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496883799
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
+
+ if hostname:
+ # Remote RPC
+ if rpc_key:
+ logger.debug("running on remote RPC tracker with key %s", rpc_key)
+ remote = request_remote(rpc_key, hostname, port, timeout=1000)
+ else:
+ logger.debug("running on remote RPC with no key")
+ remote = rpc.connect(hostname, port)
+ else:
+ # Local
+ logger.debug("running a local session")
+ remote = rpc.LocalSession()
+
+ remote.upload(os.path.join(tmp_dir, "mod.so"))
+ remote_lib = remote.load_module("mod.so")
+
+ # TODO expand to other supported devices, as listed in tvm.rpc.client (@leandron)
+ logger.debug("remote device is %s", device)
+ ctx = remote.cpu() if device == "cpu" else remote.gpu()
+
+ if profile:
+ logger.debug("creating runtime with profiling enabled")
+ module = debug_runtime.create(graph, remote_lib, ctx, dump_root="./prof")
+ else:
+ logger.debug("creating runtime with profiling disabled")
+ module = runtime.create(graph, remote_lib, ctx)
+
+ logger.debug("load params into the runtime module")
+ module.load_params(params)
+
+ shape_dict, dtype_dict = get_input_info(graph, params)
+ inputs_dict = make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode)
+
+ logger.debug("setting inputs to the module")
+ module.set_input(**inputs_dict)
+
+ # Run must be called explicitly if profiling
+ if profile:
+ logger.debug("running the module with profiling enabled")
+ module.run()
+
+ # create the module time evaluator (returns a function)
+ timer = module.module.time_evaluator("run", ctx, 1, repeat=repeat)
+ # call the evaluator function to invoke the module and save execution times
+ prof_result = timer()
+ # collect a list of execution times from the profiling results
+ times = prof_result.results
+
+ logger.debug("collecting the output tensors")
+ num_outputs = module.get_num_outputs()
+ outputs = {}
+ for i in range(num_outputs):
+ output_name = "output_{}".format(i)
+ outputs[output_name] = module.get_output(i).asnumpy()
+
+ return outputs, times
+
+
+def get_top_results(outputs, max_results):
+ """Return the top n results from the output tensor.
+
+ This function is primarily for image classification and will
+ not necessarily generalise.
+
+ Parameters
+ ----------
+ outputs : dict
+ Outputs dictionary - {output_name: np.array}.
+ max_results : int
+ Number of results to return
+
+ Returns
+ -------
+ top_results : np.array
+ Results array of shape (2, n).
+ The first row is the indices and the second is the values.
Review comment:
This feature is specifically for debugging image networks really, which happens to be very useful for us. The first row is the indices of the predicted objects (eg. from the imagenet classes) and the second row is the classification probability. That's why this is '2'. It allows you to very quickly check the sanity of an image classification network without having to dig into the output.npz.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496755746
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
Review comment:
Can we detect in Python, whether it is enabled and provide a meaningful error message?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496755746
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
Review comment:
Can we detect in Python, whether "debug graph runtime" is enabled. tp provide a meaningful error message?
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
Review comment:
Can we detect in Python, whether "debug graph runtime" is enabled. to provide a meaningful error message?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498325492
##########
File path: tests/python/driver/tvmc/conftest.py
##########
@@ -110,10 +112,33 @@ def onnx_resnet50():
@pytest.fixture(scope="session")
def tflite_compiled_module_as_tarfile(tmpdir_factory):
+
+ # Not all CI environments will have TFLite installed
+ # so we need to safely skip this fixture that will
+ # crash the tests that rely on it.
+ # As this is a pytest.fixture, we cannot take advantage
+ # of pytest.importorskip. Using the block below instead.
+ try:
+ import tflite
+ except ImportError:
+ print("Cannot import tflite, which is required by tflite_compiled_module_as_tarfile.")
+ return ""
+
target_dir = tmpdir_factory.mktemp("data")
graph, lib, params, _ = get_sample_compiled_module(target_dir)
Review comment:
Fine for me. @FrozenGene PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] mbaret commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
mbaret commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496880495
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
Review comment:
+1 for keeping these separate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496727105
##########
File path: python/tvm/driver/tvmc/common.py
##########
@@ -17,10 +17,19 @@
"""
Common utility functions shared by TVMC modules.
"""
+import logging
+
+from urllib.parse import urlparse
+
from tvm import relay
from tvm import transform
+# TODO migrate compiler logger to the common logger (@leandron)
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
Review comment:
Done. I suppose you mean #6577.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] mbaret commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
mbaret commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498237623
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
+
+ if hostname:
+ # Remote RPC
+ if rpc_key:
+ logger.debug("running on remote RPC tracker with key %s", rpc_key)
+ remote = request_remote(rpc_key, hostname, port, timeout=1000)
+ else:
+ logger.debug("running on remote RPC with no key")
+ remote = rpc.connect(hostname, port)
+ else:
+ # Local
+ logger.debug("running a local session")
+ remote = rpc.LocalSession()
+
+ remote.upload(os.path.join(tmp_dir, "mod.so"))
+ remote_lib = remote.load_module("mod.so")
+
+ # TODO expand to other supported devices, as listed in tvm.rpc.client (@leandron)
+ logger.debug("remote device is %s", device)
+ ctx = remote.cpu() if device == "cpu" else remote.gpu()
+
+ if profile:
+ logger.debug("creating runtime with profiling enabled")
+ module = debug_runtime.create(graph, remote_lib, ctx, dump_root="./prof")
+ else:
+ logger.debug("creating runtime with profiling disabled")
+ module = runtime.create(graph, remote_lib, ctx)
+
+ logger.debug("load params into the runtime module")
+ module.load_params(params)
+
+ shape_dict, dtype_dict = get_input_info(graph, params)
+ inputs_dict = make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode)
+
+ logger.debug("setting inputs to the module")
+ module.set_input(**inputs_dict)
+
+ # Run must be called explicitly if profiling
+ if profile:
+ logger.debug("running the module with profiling enabled")
+ module.run()
+
+ # create the module time evaluator (returns a function)
+ timer = module.module.time_evaluator("run", ctx, 1, repeat=repeat)
+ # call the evaluator function to invoke the module and save execution times
+ prof_result = timer()
+ # collect a list of execution times from the profiling results
+ times = prof_result.results
+
+ logger.debug("collecting the output tensors")
+ num_outputs = module.get_num_outputs()
+ outputs = {}
+ for i in range(num_outputs):
+ output_name = "output_{}".format(i)
+ outputs[output_name] = module.get_output(i).asnumpy()
+
+ return outputs, times
+
+
+def get_top_results(outputs, max_results):
+ """Return the top n results from the output tensor.
+
+ This function is primarily for image classification and will
+ not necessarily generalise.
+
+ Parameters
+ ----------
+ outputs : dict
+ Outputs dictionary - {output_name: np.array}.
+ max_results : int
+ Number of results to return
+
+ Returns
+ -------
+ top_results : np.array
+ Results array of shape (2, n).
+ The first row is the indices and the second is the values.
Review comment:
This particular feature only works for image classification networks as it was a useful shortcut for testing them. Maybe we could make that explicit in the help text?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496767403
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
Review comment:
https://github.com/apache/incubator-tvm/blob/master/python/tvm/contrib/debugger/debug_runtime.py#L67 has helped for you. But I think you could mention it to users explictly in the help text we should build it debug graph runtime firstly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac commented on pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
comaniac commented on pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#issuecomment-702493063
Thanks @leandron @FrozenGene @zhiics @mbaret @tqchen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496800421
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
Review comment:
Thanks for clarifing. The tar file is one collection, not the file of export_library
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498388155
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
+ "making it take longer to be generated.",
+ )
+ parser.add_argument(
+ "--repeat", metavar="N", type=int, default=1, help="repeat the run n times. Defaults to '1'"
+ )
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs_file=args.inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ .. note::
+ We can't simply get the input tensors from a TVM graph
+ because weight tensors are treated equivalently. Therefore, to
+ find the input tensors we look at the 'arg_nodes' in the graph
+ (which are either weights or inputs) and check which ones don't
+ appear in the params (where the weights are stored). These nodes
+ are therefore inferred to be input tensors.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(128, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs_file, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs_file : str
+ Path to a .npz file containing the inputs.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ try:
+ inputs = np.load(inputs_file) if inputs_file else {}
+ except IOError as ex:
+ raise TVMCException("Error loading inputs file: %s" % ex)
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException(
+ "the input tensor '{}' is not in the graph. Expected inputs: '{}'".format(
+ input_name, shape_dict.keys()
+ )
+ )
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs_file=None,
+ fill_mode="random",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs_file : str, optional
+ Path to an .npz file containing the inputs.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
Review comment:
https://github.com/apache/incubator-tvm/pull/6578/commits/80c36da00a8cea289ca562e7d9d87d23bedb578e
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498214578
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496784763
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
Review comment:
When you say "both .tar and .so", is it clear that inside the .tar there is a `.so` (together with the `graph` and also the `params`? Can you clarify what is the difference?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac merged pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
comaniac merged pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496771378
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
Review comment:
The practical issue moving the fill logic here is that we need the shapes of all inputs, and at this point we didn't load the model. Once we have all this information, we actually have the logic within `make_inputs_dict`.
I suggest we keep it this way, because it simplifies a bit for testing, having `run_module` with an `input` parameter, that allows us to inject inputs. If you feel the design is too bad, I can try to find alternatives.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498385862
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
+ "making it take longer to be generated.",
+ )
+ parser.add_argument(
+ "--repeat", metavar="N", type=int, default=1, help="repeat the run n times. Defaults to '1'"
+ )
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
Review comment:
This is mostly to keep he convention other `tvmc` subcommands are using. It was more like a design decision, to mark the artefact in which the subcommand is acting on. Other tools, for example `clang` and `gcc` use similar convention.
(please note that I don't want to pretend being a specialist in command line tools, and I'm very happy with all the feeback I've been receiving :slightly_smiling_face: )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496781556
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
Review comment:
fixed
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498143609
##########
File path: tests/python/driver/tvmc/conftest.py
##########
@@ -110,10 +112,33 @@ def onnx_resnet50():
@pytest.fixture(scope="session")
def tflite_compiled_module_as_tarfile(tmpdir_factory):
+
+ # Not all CI environments will have TFLite installed
+ # so we need to safely skip this fixture that will
+ # crash the tests that rely on it.
+ # As this is a pytest.fixture, we cannot take advantage
+ # of pytest.importorskip. Using the block below instead.
+ try:
+ import tflite
+ except ImportError:
+ print("Cannot import tflite, which is required by tflite_compiled_module_as_tarfile.")
+ return ""
+
target_dir = tmpdir_factory.mktemp("data")
graph, lib, params, _ = get_sample_compiled_module(target_dir)
Review comment:
Got it. As it will be deprecated in the next release, and I find it a bit risky to push a last minute change to the compiler (before the release), I'm gonna add the TODO for now, in the interest of risk and time.
We should update this soon, in a new PR to adjust `compile` and `run`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498376833
##########
File path: tests/python/driver/tvmc/test_runner.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import pytest
+import numpy as np
+
+from tvm.driver import tvmc
+
+
+def test_generate_tensor_data_zeros():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "zeros")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data_ones():
+ expected_shape = (224, 224)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "ones")
+
+ assert sut.shape == (224, 224)
+
+
+def test_generate_tensor_data_random():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "random")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data__type_unknown():
+ with pytest.raises(tvmc.common.TVMCException) as e:
+ tvmc.runner.generate_tensor_data((2, 3), "float32", "whatever")
+
+
+def test_format_times__contains_header():
+ sut = tvmc.runner.format_times([60, 120, 12, 42])
+ assert "std (s)" in sut
+
+
+def test_get_top_results_keep_results():
+ fake_outputs = {"output_0": np.array([[1, 2, 3, 4], [5, 6, 7, 8]])}
+ number_of_results_wanted = 3
+ sut = tvmc.runner.get_top_results(fake_outputs, number_of_results_wanted)
+
+ expected_number_of_lines = 2
+ assert len(sut) == expected_number_of_lines
+
+ expected_number_of_results_per_line = 3
+ assert len(sut[0]) == expected_number_of_results_per_line
+ assert len(sut[1]) == expected_number_of_results_per_line
+
+
+def test_run_tflite_module__with_profile__valid_input(
Review comment:
why there are many function names using double underscore?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r497676113
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
Review comment:
I understand your concern. Then how about moving this logic to `make_inputs_dict`? Like simply passing`args.inputs` all the way to `make_inputs_dict` and call `np.load` there.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
+
+ if hostname:
+ # Remote RPC
+ if rpc_key:
+ logger.debug("running on remote RPC tracker with key %s", rpc_key)
+ remote = request_remote(rpc_key, hostname, port, timeout=1000)
+ else:
+ logger.debug("running on remote RPC with no key")
+ remote = rpc.connect(hostname, port)
+ else:
+ # Local
+ logger.debug("running a local session")
+ remote = rpc.LocalSession()
+
+ remote.upload(os.path.join(tmp_dir, "mod.so"))
+ remote_lib = remote.load_module("mod.so")
+
+ # TODO expand to other supported devices, as listed in tvm.rpc.client (@leandron)
+ logger.debug("remote device is %s", device)
+ ctx = remote.cpu() if device == "cpu" else remote.gpu()
+
+ if profile:
+ logger.debug("creating runtime with profiling enabled")
+ module = debug_runtime.create(graph, remote_lib, ctx, dump_root="./prof")
+ else:
+ logger.debug("creating runtime with profiling disabled")
+ module = runtime.create(graph, remote_lib, ctx)
+
+ logger.debug("load params into the runtime module")
+ module.load_params(params)
+
+ shape_dict, dtype_dict = get_input_info(graph, params)
+ inputs_dict = make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode)
+
+ logger.debug("setting inputs to the module")
+ module.set_input(**inputs_dict)
+
+ # Run must be called explicitly if profiling
+ if profile:
+ logger.debug("running the module with profiling enabled")
+ module.run()
+
+ # create the module time evaluator (returns a function)
+ timer = module.module.time_evaluator("run", ctx, 1, repeat=repeat)
+ # call the evaluator function to invoke the module and save execution times
+ prof_result = timer()
+ # collect a list of execution times from the profiling results
+ times = prof_result.results
+
+ logger.debug("collecting the output tensors")
+ num_outputs = module.get_num_outputs()
+ outputs = {}
+ for i in range(num_outputs):
+ output_name = "output_{}".format(i)
+ outputs[output_name] = module.get_output(i).asnumpy()
+
+ return outputs, times
+
+
+def get_top_results(outputs, max_results):
+ """Return the top n results from the output tensor.
+
+ This function is primarily for image classification and will
+ not necessarily generalise.
+
+ Parameters
+ ----------
+ outputs : dict
+ Outputs dictionary - {output_name: np.array}.
+ max_results : int
+ Number of results to return
+
+ Returns
+ -------
+ top_results : np.array
+ Results array of shape (2, n).
+ The first row is the indices and the second is the values.
Review comment:
Thanks for the clarification. IIUC, it seems like we limit the applications to be CNN models and do not consider others like object detection or NLP models? If so please add the descriptions in the corresponding places.
##########
File path: tests/python/driver/tvmc/conftest.py
##########
@@ -110,10 +112,33 @@ def onnx_resnet50():
@pytest.fixture(scope="session")
def tflite_compiled_module_as_tarfile(tmpdir_factory):
+
+ # Not all CI environments will have TFLite installed
+ # so we need to safely skip this fixture that will
+ # crash the tests that rely on it.
+ # As this is a pytest.fixture, we cannot take advantage
+ # of pytest.importorskip. Using the block below instead.
+ try:
+ import tflite
+ except ImportError:
+ print("Cannot import tflite, which is required by tflite_compiled_module_as_tarfile.")
+ return ""
+
target_dir = tmpdir_factory.mktemp("data")
graph, lib, params, _ = get_sample_compiled_module(target_dir)
Review comment:
I agree with @FrozenGene that we should use the up-to-date APIs. @leandron please fix it in this PR or add a TODO to fix it in a follow-up PR.
##########
File path: tests/python/driver/tvmc/test_runner.py
##########
@@ -0,0 +1,114 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import pytest
+import numpy as np
+
+from tvm.driver import tvmc
+
+
+def test_generate_tensor_data_zeros():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "zeros")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data_ones():
+ expected_shape = (224, 224)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "ones")
+
+ assert sut.shape == (224, 224)
+
+
+def test_generate_tensor_data_random():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "random")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data__type_unknown():
+ with pytest.raises(tvmc.common.TVMCException) as e:
+ tvmc.runner.generate_tensor_data((2, 3), "float32", "whatever")
+
+
+def test_format_times__contains_header():
+ sut = tvmc.runner.format_times([60, 120, 12, 42])
+ assert "std (s)" in sut
+
+
+def test_get_top_results_keep_results():
+ fake_outputs = {"output_0": np.array([[1, 2, 3, 4], [5, 6, 7, 8]])}
+ number_of_results_wanted = 3
+ sut = tvmc.runner.get_top_results(fake_outputs, number_of_results_wanted)
+
+ expected_number_of_lines = 2
+ assert len(sut) == expected_number_of_lines
+
+ expected_number_of_results_per_line = 3
+ assert len(sut[0]) == expected_number_of_results_per_line
+ assert len(sut[1]) == expected_number_of_results_per_line
+
+
+def test_get_top_results_keep_results__limit_bigger_than_returned():
+ fake_outputs = {"output_0": np.array([[1, 2, 3, 4], [5, 6, 7, 8]])}
+ number_of_results_wanted = 6
+ sut = tvmc.runner.get_top_results(fake_outputs, number_of_results_wanted)
+
+ expected_number_of_lines = 2
+ assert len(sut) == expected_number_of_lines
+
+ # despite 'number_of_results_wanted' being 6,
+ # we limit to the maximum available, in this
+ # case it is 4.
Review comment:
Sure we can remove it if it does not improve the test coverage.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496214531
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
Review comment:
My understanding of the difference between `target` and `device` is:
* `--target` from both `tvmc compile` and `tvmc tune` will plug into the `Target` API, i.e. `tvm.target.Target(target)` and accepts things like `llvm ...` or ` cuda...` or `c ...`
* `--device`, which I think only belongs to `tvmc run` is connected to a `session`, as in `tvm.rpc.client.RPCSession.*`, where * can be `.cpu()`, `.gpu()`, `.cl()`, `.webgpu()`, etc.
Again, my understanding is the it doesn't make sense `tvmc compile --target cpu` or `tvmc run --target llvm`. This is where I think expectations from an unified `--target` argument would be misleading.
OTOH, in case you think it makes sense or I'm getting something wrong, please let me know.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498143767
##########
File path: tests/python/driver/tvmc/test_runner.py
##########
@@ -0,0 +1,114 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import pytest
+import numpy as np
+
+from tvm.driver import tvmc
+
+
+def test_generate_tensor_data_zeros():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "zeros")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data_ones():
+ expected_shape = (224, 224)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "ones")
+
+ assert sut.shape == (224, 224)
+
+
+def test_generate_tensor_data_random():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "random")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data__type_unknown():
+ with pytest.raises(tvmc.common.TVMCException) as e:
+ tvmc.runner.generate_tensor_data((2, 3), "float32", "whatever")
+
+
+def test_format_times__contains_header():
+ sut = tvmc.runner.format_times([60, 120, 12, 42])
+ assert "std (s)" in sut
+
+
+def test_get_top_results_keep_results():
+ fake_outputs = {"output_0": np.array([[1, 2, 3, 4], [5, 6, 7, 8]])}
+ number_of_results_wanted = 3
+ sut = tvmc.runner.get_top_results(fake_outputs, number_of_results_wanted)
+
+ expected_number_of_lines = 2
+ assert len(sut) == expected_number_of_lines
+
+ expected_number_of_results_per_line = 3
+ assert len(sut[0]) == expected_number_of_results_per_line
+ assert len(sut[1]) == expected_number_of_results_per_line
+
+
+def test_get_top_results_keep_results__limit_bigger_than_returned():
+ fake_outputs = {"output_0": np.array([[1, 2, 3, 4], [5, 6, 7, 8]])}
+ number_of_results_wanted = 6
+ sut = tvmc.runner.get_top_results(fake_outputs, number_of_results_wanted)
+
+ expected_number_of_lines = 2
+ assert len(sut) == expected_number_of_lines
+
+ # despite 'number_of_results_wanted' being 6,
+ # we limit to the maximum available, in this
+ # case it is 4.
Review comment:
Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498386116
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
Review comment:
That is a good question. @comaniac can you comment? Maybe this needs some rephrasing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496808406
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
Review comment:
I agree that target and device are different thing, one is a compile time concept(what to generate), another is runtime context(what GPU to run)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496225679
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
Review comment:
I see, not sure it's because I used TVM all the time so it looks fine to me lol
Let me tag others for more data @zhiics @masahi @tqchen @junrushao1994
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496205424
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
Review comment:
Hmm...that's my intention to have a consistent subcommend for the same semantic. Could you elaborate what might be confusing in the case of using `--target` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498369599
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
Review comment:
I am a little curious about this sentence. As far as I know, debug graph runtime is just to record op execution time, no other debug flags to add to build tvm(like -g). So how does it affect the inference time?
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
+ "making it take longer to be generated.",
+ )
+ parser.add_argument(
+ "--repeat", metavar="N", type=int, default=1, help="repeat the run n times. Defaults to '1'"
+ )
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
Review comment:
how about change to `—file`? Other arguments starts `—`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496731437
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
+
+ if hostname:
+ # Remote RPC
+ if rpc_key:
+ logger.debug("running on remote RPC tracker with key %s", rpc_key)
+ remote = request_remote(rpc_key, hostname, port, timeout=1000)
+ else:
+ logger.debug("running on remote RPC with no key")
+ remote = rpc.connect(hostname, port)
+ else:
+ # Local
+ logger.debug("running a local session")
+ remote = rpc.LocalSession()
Review comment:
Fixed
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
+
+ if hostname:
+ # Remote RPC
+ if rpc_key:
+ logger.debug("running on remote RPC tracker with key %s", rpc_key)
+ remote = request_remote(rpc_key, hostname, port, timeout=1000)
+ else:
+ logger.debug("running on remote RPC with no key")
+ remote = rpc.connect(hostname, port)
+ else:
+ # Local
+ logger.debug("running a local session")
+ remote = rpc.LocalSession()
+
+ remote.upload(os.path.join(tmp_dir, "mod.so"))
+ remote_lib = remote.load_module("mod.so")
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498385438
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
Review comment:
Even running debug runtime would affect the inference time compared to the graph runtime.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496732677
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
+
+ if hostname:
+ # Remote RPC
+ if rpc_key:
+ logger.debug("running on remote RPC tracker with key %s", rpc_key)
+ remote = request_remote(rpc_key, hostname, port, timeout=1000)
+ else:
+ logger.debug("running on remote RPC with no key")
+ remote = rpc.connect(hostname, port)
+ else:
+ # Local
+ logger.debug("running a local session")
+ remote = rpc.LocalSession()
+
+ remote.upload(os.path.join(tmp_dir, "mod.so"))
+ remote_lib = remote.load_module("mod.so")
+
+ # TODO expand to other supported devices, as listed in tvm.rpc.client (@leandron)
+ logger.debug("remote device is %s", device)
+ ctx = remote.cpu() if device == "cpu" else remote.gpu()
+
+ if profile:
+ logger.debug("creating runtime with profiling enabled")
+ module = debug_runtime.create(graph, remote_lib, ctx, dump_root="./prof")
+ else:
+ logger.debug("creating runtime with profiling disabled")
+ module = runtime.create(graph, remote_lib, ctx)
+
+ logger.debug("load params into the runtime module")
+ module.load_params(params)
+
+ shape_dict, dtype_dict = get_input_info(graph, params)
+ inputs_dict = make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode)
+
+ logger.debug("setting inputs to the module")
+ module.set_input(**inputs_dict)
+
+ # Run must be called explicitly if profiling
+ if profile:
+ logger.debug("running the module with profiling enabled")
+ module.run()
+
+ # create the module time evaluator (returns a function)
+ timer = module.module.time_evaluator("run", ctx, 1, repeat=repeat)
+ # call the evaluator function to invoke the module and save execution times
+ prof_result = timer()
+ # collect a list of execution times from the profiling results
+ times = prof_result.results
+
+ logger.debug("collecting the output tensors")
+ num_outputs = module.get_num_outputs()
+ outputs = {}
+ for i in range(num_outputs):
+ output_name = "output_{}".format(i)
+ outputs[output_name] = module.get_output(i).asnumpy()
+
+ return outputs, times
+
+
+def get_top_results(outputs, max_results):
+ """Return the top n results from the output tensor.
+
+ This function is primarily for image classification and will
+ not necessarily generalise.
+
+ Parameters
+ ----------
+ outputs : dict
+ Outputs dictionary - {output_name: np.array}.
+ max_results : int
+ Number of results to return
+
+ Returns
+ -------
+ top_results : np.array
+ Results array of shape (2, n).
+ The first row is the indices and the second is the values.
+
+ """
+ output = outputs["output_0"]
+ sorted_labels = output.argsort()[0][-max_results:][::-1]
+ output.sort()
+ sorted_values = output[0][-max_results:][::-1]
+ top_results = np.array([sorted_labels, sorted_values])
+ return top_results
+
+
+def format_times(times):
+ """Format the mean, max, min and std of the execution times.
+
+ Parameters
+ ----------
+ times : list
+ A list of execution times (in seconds).
+
+ Returns
+ -------
+ str
+ A formatted string containing the statistics.
+ """
+ # This has the effect of producing a small table that looks like:
+ #
+ # Execution time summary:
+ # mean (s) max (s) min (s) std (s)
+ # 0.14310 0.16161 0.12933 0.01004
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496830062
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
Review comment:
I struggled to understand it when I read it first. But after the discussion with @FrozenGene, I don't think we can use this API, because we have our own `.tar` contents, generated by `tvmc compile`, that is not and `.o` file to be passed to the compiler.
I hope I got it right, let me know in case I'm missing something.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498366952
##########
File path: tests/python/driver/tvmc/conftest.py
##########
@@ -110,10 +112,33 @@ def onnx_resnet50():
@pytest.fixture(scope="session")
def tflite_compiled_module_as_tarfile(tmpdir_factory):
+
+ # Not all CI environments will have TFLite installed
+ # so we need to safely skip this fixture that will
+ # crash the tests that rely on it.
+ # As this is a pytest.fixture, we cannot take advantage
+ # of pytest.importorskip. Using the block below instead.
+ try:
+ import tflite
+ except ImportError:
+ print("Cannot import tflite, which is required by tflite_compiled_module_as_tarfile.")
+ return ""
+
target_dir = tmpdir_factory.mktemp("data")
graph, lib, params, _ = get_sample_compiled_module(target_dir)
Review comment:
Add one TODO is acceptable for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496781713
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498563406
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
Review comment:
However, this overhead shouldn’t affect the inferece time in my opinion. Because we record this using chrono, when we dump, the time is recorded completely and shouldn’t affect our record.
If we want to describle the overhead, maybe we shouldn’t use the inference time and to say we will introduce the overload time of collecting / dumping which will make our pipiline longer than before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498565625
##########
File path: tests/python/driver/tvmc/test_runner.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import pytest
+import numpy as np
+
+from tvm.driver import tvmc
+
+
+def test_generate_tensor_data_zeros():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "zeros")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data_ones():
+ expected_shape = (224, 224)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "ones")
+
+ assert sut.shape == (224, 224)
+
+
+def test_generate_tensor_data_random():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "random")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data__type_unknown():
+ with pytest.raises(tvmc.common.TVMCException) as e:
+ tvmc.runner.generate_tensor_data((2, 3), "float32", "whatever")
+
+
+def test_format_times__contains_header():
+ sut = tvmc.runner.format_times([60, 120, 12, 42])
+ assert "std (s)" in sut
+
+
+def test_get_top_results_keep_results():
+ fake_outputs = {"output_0": np.array([[1, 2, 3, 4], [5, 6, 7, 8]])}
+ number_of_results_wanted = 3
+ sut = tvmc.runner.get_top_results(fake_outputs, number_of_results_wanted)
+
+ expected_number_of_lines = 2
+ assert len(sut) == expected_number_of_lines
+
+ expected_number_of_results_per_line = 3
+ assert len(sut[0]) == expected_number_of_results_per_line
+ assert len(sut[1]) == expected_number_of_results_per_line
+
+
+def test_run_tflite_module__with_profile__valid_input(
Review comment:
I suggest we keeping the style like all other test. Current function names are like generated by automatic tools. How about this suggestion?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498374296
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
+ "making it take longer to be generated.",
+ )
+ parser.add_argument(
+ "--repeat", metavar="N", type=int, default=1, help="repeat the run n times. Defaults to '1'"
+ )
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs_file=args.inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ .. note::
+ We can't simply get the input tensors from a TVM graph
+ because weight tensors are treated equivalently. Therefore, to
+ find the input tensors we look at the 'arg_nodes' in the graph
+ (which are either weights or inputs) and check which ones don't
+ appear in the params (where the weights are stored). These nodes
+ are therefore inferred to be input tensors.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(128, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs_file, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs_file : str
+ Path to a .npz file containing the inputs.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ try:
+ inputs = np.load(inputs_file) if inputs_file else {}
+ except IOError as ex:
+ raise TVMCException("Error loading inputs file: %s" % ex)
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException(
+ "the input tensor '{}' is not in the graph. Expected inputs: '{}'".format(
+ input_name, shape_dict.keys()
+ )
+ )
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs_file=None,
+ fill_mode="random",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs_file : str, optional
+ Path to an .npz file containing the inputs.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
Review comment:
random.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] mbaret commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
mbaret commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496881232
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
Review comment:
Maybe we could say 'fill all unspecified input tensors with values'?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498143609
##########
File path: tests/python/driver/tvmc/conftest.py
##########
@@ -110,10 +112,33 @@ def onnx_resnet50():
@pytest.fixture(scope="session")
def tflite_compiled_module_as_tarfile(tmpdir_factory):
+
+ # Not all CI environments will have TFLite installed
+ # so we need to safely skip this fixture that will
+ # crash the tests that rely on it.
+ # As this is a pytest.fixture, we cannot take advantage
+ # of pytest.importorskip. Using the block below instead.
+ try:
+ import tflite
+ except ImportError:
+ print("Cannot import tflite, which is required by tflite_compiled_module_as_tarfile.")
+ return ""
+
target_dir = tmpdir_factory.mktemp("data")
graph, lib, params, _ = get_sample_compiled_module(target_dir)
Review comment:
Got it. As it will be deprecated in the next release, and I find it a bit risky to push a last minute change to the compiler (before the release), I'm gonna add the TODO for now, in the interest of risk and time.
We should soon update this soon, in a new PR to adjust `compile` and `run`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496788964
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
Review comment:
My meaning is when we compile, we could let users `lib.export_library`, which supports two formats: .tar and .so. If it is .tar, which has .o inside and will require remote devices extract tar files and call compiler to compile it into .so. If it is .so, remote devices only needs runtime and could load it directly, no need to extract tar files and has compiler.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496808406
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
Review comment:
I agree that target and device are different thing, one is a compile time concept(what to generate), another is runtime context(what GPU to run). To clarify, perhaps we could do --runtime-ctx
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r497209168
##########
File path: tests/python/driver/tvmc/conftest.py
##########
@@ -110,10 +112,33 @@ def onnx_resnet50():
@pytest.fixture(scope="session")
def tflite_compiled_module_as_tarfile(tmpdir_factory):
+
+ # Not all CI environments will have TFLite installed
+ # so we need to safely skip this fixture that will
+ # crash the tests that rely on it.
+ # As this is a pytest.fixture, we cannot take advantage
+ # of pytest.importorskip. Using the block below instead.
+ try:
+ import tflite
+ except ImportError:
+ print("Cannot import tflite, which is required by tflite_compiled_module_as_tarfile.")
+ return ""
+
target_dir = tmpdir_factory.mktemp("data")
graph, lib, params, _ = get_sample_compiled_module(target_dir)
Review comment:
Just find one very important place here. Do we still use old api of tvm.build? In new module-based api, we will only generate one `lib`, the `graph` and `params` is serialized into `lib`. Of course, we will still have one interface to make users get `params` separately to meet the need of network only updates weight only. We should notice that old api will be removed in the next release, so we shouldn't use it in the new feature.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498489041
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
Review comment:
Yeah I'm just saying the overhead of collecting/dumping op execution time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498386116
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
Review comment:
That is a good question. @comaniac can you comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496142487
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
Review comment:
This logic is okay to me for now. In the long run, we may want to change graph runtime a bit and let it save inputs and params separately. params can be obtained from MetadataModule directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498386079
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
+ "making it take longer to be generated.",
+ )
+ parser.add_argument(
+ "--repeat", metavar="N", type=int, default=1, help="repeat the run n times. Defaults to '1'"
+ )
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
Review comment:
The argument without dashes are positional arguments. This behavior is same as most other compilers like `gcc <options> FILE`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r497301769
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
Review comment:
I tried to clarify this in the `--help` option, that is in this commit: https://github.com/apache/incubator-tvm/pull/6578/commits/4b307643fc2aeabedcda49873c4d57d1be88349c
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496740342
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
Review comment:
fill zeros is not a good default value. let us fill random values by default.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
Review comment:
let us restrict to 128. If the model is int8 (not uint8). 256 will make us overflow.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
Review comment:
let us support both mode (tar and .so). Because not all remote devices has compiler.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
Review comment:
I think we should mention users need to turn on debug graph runtime in the config.cmake.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
Review comment:
Same comment as before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496201533
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
Review comment:
I took some time thinking about this option, and playing with it on my machine. I concluded that users might get confused and have expectations if we use `—target` terminology here, as we have it in other subcommands.
This is why I decided to go with the terminology used by the underlying APIs, and call it device instead. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496846703
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
Review comment:
Inputs coming from `--input` will take precedence. In case your model asks for an input that is not in the inputs, it will be filled using `--fill-mode`.
We don't really expect it to be used this way, but will make help clearer about what is the expected behaviour..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r497362221
##########
File path: tests/python/driver/tvmc/conftest.py
##########
@@ -110,10 +112,33 @@ def onnx_resnet50():
@pytest.fixture(scope="session")
def tflite_compiled_module_as_tarfile(tmpdir_factory):
+
+ # Not all CI environments will have TFLite installed
+ # so we need to safely skip this fixture that will
+ # crash the tests that rely on it.
+ # As this is a pytest.fixture, we cannot take advantage
+ # of pytest.importorskip. Using the block below instead.
+ try:
+ import tflite
+ except ImportError:
+ print("Cannot import tflite, which is required by tflite_compiled_module_as_tarfile.")
+ return ""
+
target_dir = tmpdir_factory.mktemp("data")
graph, lib, params, _ = get_sample_compiled_module(target_dir)
Review comment:
My meaning is here:
https://github.com/apache/incubator-tvm/blob/fde8d432acd2d7964607c4f9216720298db04007/python/tvm/driver/tvmc/compiler.py#L215
We shouldn't rely on these `getter`, because this is just for old api compatibility and will remove them (except get params) in the next release. We should support to run the `lib` directly. If not in this pr, we should implement it in the next pr.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496830062
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
Review comment:
I struggled to understand it when I read it first. But after the discussion with @FrozenGene, I don't think we can use this API, because we have our own `.tar` contents, generated by `tvmc compile`, that is not a `.o` file to be passed to the compiler.
I hope I got it right, let me know in case I'm missing something.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496112356
##########
File path: python/tvm/driver/tvmc/common.py
##########
@@ -17,10 +17,19 @@
"""
Common utility functions shared by TVMC modules.
"""
+import logging
+
+from urllib.parse import urlparse
+
from tvm import relay
from tvm import transform
+# TODO migrate compiler logger to the common logger (@leandron)
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
Review comment:
Please rebase and resolve this TODO after #6557 is merged.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
Review comment:
Why not just `target`? Suppose users have use `compile` to build a model, then it should be straightforward to use the same target here.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
+
+ if hostname:
+ # Remote RPC
+ if rpc_key:
+ logger.debug("running on remote RPC tracker with key %s", rpc_key)
+ remote = request_remote(rpc_key, hostname, port, timeout=1000)
+ else:
+ logger.debug("running on remote RPC with no key")
+ remote = rpc.connect(hostname, port)
+ else:
+ # Local
+ logger.debug("running a local session")
+ remote = rpc.LocalSession()
Review comment:
Given that `remote` could be a local session, we should use a more general name, such as `session`.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
+
+ if hostname:
+ # Remote RPC
+ if rpc_key:
+ logger.debug("running on remote RPC tracker with key %s", rpc_key)
+ remote = request_remote(rpc_key, hostname, port, timeout=1000)
+ else:
+ logger.debug("running on remote RPC with no key")
+ remote = rpc.connect(hostname, port)
+ else:
+ # Local
+ logger.debug("running a local session")
+ remote = rpc.LocalSession()
+
+ remote.upload(os.path.join(tmp_dir, "mod.so"))
+ remote_lib = remote.load_module("mod.so")
+
+ # TODO expand to other supported devices, as listed in tvm.rpc.client (@leandron)
+ logger.debug("remote device is %s", device)
+ ctx = remote.cpu() if device == "cpu" else remote.gpu()
+
+ if profile:
+ logger.debug("creating runtime with profiling enabled")
+ module = debug_runtime.create(graph, remote_lib, ctx, dump_root="./prof")
+ else:
+ logger.debug("creating runtime with profiling disabled")
+ module = runtime.create(graph, remote_lib, ctx)
+
+ logger.debug("load params into the runtime module")
+ module.load_params(params)
+
+ shape_dict, dtype_dict = get_input_info(graph, params)
+ inputs_dict = make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode)
+
+ logger.debug("setting inputs to the module")
+ module.set_input(**inputs_dict)
+
+ # Run must be called explicitly if profiling
+ if profile:
+ logger.debug("running the module with profiling enabled")
+ module.run()
+
+ # create the module time evaluator (returns a function)
+ timer = module.module.time_evaluator("run", ctx, 1, repeat=repeat)
+ # call the evaluator function to invoke the module and save execution times
+ prof_result = timer()
+ # collect a list of execution times from the profiling results
+ times = prof_result.results
+
+ logger.debug("collecting the output tensors")
+ num_outputs = module.get_num_outputs()
+ outputs = {}
+ for i in range(num_outputs):
+ output_name = "output_{}".format(i)
+ outputs[output_name] = module.get_output(i).asnumpy()
+
+ return outputs, times
+
+
+def get_top_results(outputs, max_results):
+ """Return the top n results from the output tensor.
+
+ This function is primarily for image classification and will
+ not necessarily generalise.
+
+ Parameters
+ ----------
+ outputs : dict
+ Outputs dictionary - {output_name: np.array}.
+ max_results : int
+ Number of results to return
+
+ Returns
+ -------
+ top_results : np.array
+ Results array of shape (2, n).
+ The first row is the indices and the second is the values.
Review comment:
Why it is always "2"?
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
+
+ if hostname:
+ # Remote RPC
+ if rpc_key:
+ logger.debug("running on remote RPC tracker with key %s", rpc_key)
+ remote = request_remote(rpc_key, hostname, port, timeout=1000)
+ else:
+ logger.debug("running on remote RPC with no key")
+ remote = rpc.connect(hostname, port)
+ else:
+ # Local
+ logger.debug("running a local session")
+ remote = rpc.LocalSession()
+
+ remote.upload(os.path.join(tmp_dir, "mod.so"))
+ remote_lib = remote.load_module("mod.so")
+
+ # TODO expand to other supported devices, as listed in tvm.rpc.client (@leandron)
+ logger.debug("remote device is %s", device)
+ ctx = remote.cpu() if device == "cpu" else remote.gpu()
+
+ if profile:
+ logger.debug("creating runtime with profiling enabled")
+ module = debug_runtime.create(graph, remote_lib, ctx, dump_root="./prof")
+ else:
+ logger.debug("creating runtime with profiling disabled")
+ module = runtime.create(graph, remote_lib, ctx)
+
+ logger.debug("load params into the runtime module")
+ module.load_params(params)
+
+ shape_dict, dtype_dict = get_input_info(graph, params)
+ inputs_dict = make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode)
+
+ logger.debug("setting inputs to the module")
+ module.set_input(**inputs_dict)
+
+ # Run must be called explicitly if profiling
+ if profile:
+ logger.debug("running the module with profiling enabled")
+ module.run()
+
+ # create the module time evaluator (returns a function)
+ timer = module.module.time_evaluator("run", ctx, 1, repeat=repeat)
+ # call the evaluator function to invoke the module and save execution times
+ prof_result = timer()
+ # collect a list of execution times from the profiling results
+ times = prof_result.results
+
+ logger.debug("collecting the output tensors")
+ num_outputs = module.get_num_outputs()
+ outputs = {}
+ for i in range(num_outputs):
+ output_name = "output_{}".format(i)
+ outputs[output_name] = module.get_output(i).asnumpy()
+
+ return outputs, times
+
+
+def get_top_results(outputs, max_results):
+ """Return the top n results from the output tensor.
+
+ This function is primarily for image classification and will
+ not necessarily generalise.
+
+ Parameters
+ ----------
+ outputs : dict
+ Outputs dictionary - {output_name: np.array}.
+ max_results : int
+ Number of results to return
+
+ Returns
+ -------
+ top_results : np.array
+ Results array of shape (2, n).
+ The first row is the indices and the second is the values.
+
+ """
+ output = outputs["output_0"]
+ sorted_labels = output.argsort()[0][-max_results:][::-1]
+ output.sort()
+ sorted_values = output[0][-max_results:][::-1]
+ top_results = np.array([sorted_labels, sorted_values])
+ return top_results
+
+
+def format_times(times):
+ """Format the mean, max, min and std of the execution times.
+
+ Parameters
+ ----------
+ times : list
+ A list of execution times (in seconds).
+
+ Returns
+ -------
+ str
+ A formatted string containing the statistics.
+ """
+ # This has the effect of producing a small table that looks like:
+ #
+ # Execution time summary:
+ # mean (s) max (s) min (s) std (s)
+ # 0.14310 0.16161 0.12933 0.01004
Review comment:
Better to put this into the docstring as a note so that people can easily see it in the API reference without looking into the code.
##########
File path: python/tvm/driver/tvmc/common.py
##########
@@ -63,3 +72,32 @@ def convert_graph_layout(mod, desired_layout):
raise TVMCException(
"Error converting layout to {0}: {1}".format(desired_layout, str(err))
)
+
+
+def tracker_host_port_from_cli(rpc_tracker_str):
+ """Alter the layout of the input graph.
Review comment:
Wrong desc.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
Review comment:
IIUC, this will use debug runtime. Then it might be better to mention this may affect the inference performance.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
Review comment:
What's the expected semantic when both `--fill-mode` and `-i` are specified? Need to mention it in the help.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
Review comment:
Better to integrate the logic of filling mode to here so that we can have a unified place to deal with input tensors.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
Review comment:
Can we use the built-in APIs to deal with module loading?
Similar to in https://tvm.apache.org/docs/tutorials/frontend/deploy_model_on_rasp.html#deploy-the-model-remotely-by-rpc
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
Review comment:
- Put this note to the docstring, like https://github.com/apache/incubator-tvm/blob/master/python/tvm/relay/op/transform.py#L414
- cc @zhiics @masahi to help check this logic.
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
+ t.extractall(tmp_dir)
+ graph = open(os.path.join(tmp_dir, "mod.json")).read()
+ params = bytearray(open(os.path.join(tmp_dir, "mod.params"), "rb").read())
+
+ if hostname:
+ # Remote RPC
+ if rpc_key:
+ logger.debug("running on remote RPC tracker with key %s", rpc_key)
+ remote = request_remote(rpc_key, hostname, port, timeout=1000)
+ else:
+ logger.debug("running on remote RPC with no key")
+ remote = rpc.connect(hostname, port)
+ else:
+ # Local
+ logger.debug("running a local session")
+ remote = rpc.LocalSession()
+
+ remote.upload(os.path.join(tmp_dir, "mod.so"))
+ remote_lib = remote.load_module("mod.so")
Review comment:
Given that `remote_lib` may not be "remote", we should use a more general name, such as `lib`.
##########
File path: tests/python/driver/tvmc/test_runner.py
##########
@@ -0,0 +1,114 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import pytest
+import numpy as np
+
+from tvm.driver import tvmc
+
+
+def test_generate_tensor_data_zeros():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "zeros")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data_ones():
+ expected_shape = (224, 224)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "ones")
+
+ assert sut.shape == (224, 224)
+
+
+def test_generate_tensor_data_random():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "random")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data__type_unknown():
+ with pytest.raises(tvmc.common.TVMCException) as e:
+ tvmc.runner.generate_tensor_data((2, 3), "float32", "whatever")
+
+
+def test_format_times__contains_header():
+ sut = tvmc.runner.format_times([60, 120, 12, 42])
+ assert "std (s)" in sut
+
+
+def test_get_top_results_keep_results():
+ fake_outputs = {"output_0": np.array([[1, 2, 3, 4], [5, 6, 7, 8]])}
+ number_of_results_wanted = 3
+ sut = tvmc.runner.get_top_results(fake_outputs, number_of_results_wanted)
+
+ expected_number_of_lines = 2
+ assert len(sut) == expected_number_of_lines
+
+ expected_number_of_results_per_line = 3
+ assert len(sut[0]) == expected_number_of_results_per_line
+ assert len(sut[1]) == expected_number_of_results_per_line
+
+
+def test_get_top_results_keep_results__limit_bigger_than_returned():
+ fake_outputs = {"output_0": np.array([[1, 2, 3, 4], [5, 6, 7, 8]])}
+ number_of_results_wanted = 6
+ sut = tvmc.runner.get_top_results(fake_outputs, number_of_results_wanted)
+
+ expected_number_of_lines = 2
+ assert len(sut) == expected_number_of_lines
+
+ # despite 'number_of_results_wanted' being 6,
+ # we limit to the maximum available, in this
+ # case it is 4.
Review comment:
I didn't find the corresponding logic. Could you point out?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496796368
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(256, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs : dict
+ Input data dictionary - {input_name: np.array}.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException("the input tensor '{}' is not in the graph".format(input_name))
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs=None,
+ fill_mode="zeros",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs : dict, optional
+ A dictionary of {input_name: np.array} storing the input
+ tensors to the network. If this is not specified, data will
+ be automatically generated for the input tensors.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
+ repeat : int, optional
+ How many times to repeat the run.
+ profile : bool
+ Whether to profile the run with the debug runtime.
+
+ Returns
+ -------
+ outputs : dict
+ a dictionary with output tensors, generated by the module
+ times : list of str
+ execution times generated by the time evaluator
+ """
+ if not inputs:
+ inputs = {}
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ logger.debug("extracting module file %s", module_file)
+ t = tarfile.open(module_file)
Review comment:
Oh, actually then you run `tvmc compile` we create that tar manually, with the `.so` inside. Then I think we are covered for that specific `.so` case.
https://github.com/apache/incubator-tvm/blob/master/python/tvm/driver/tvmc/compiler.py#L218
Let me know if there is something else I'm missing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498566208
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
Review comment:
Yeah, of course.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498383337
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,464 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module. Defaults to 'cpu'",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="random",
+ help="fill all input tensors with values. In case --inputs/-i is provided, "
+ "they will take precedence over --fill-mode. Any remaining inputs will be "
+ "filled using the chosen fill mode. Defaults to 'random'",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile",
+ action="store_true",
+ help="generate profiling data from the runtime execution. "
+ "Using --profile requires the Graph Runtime Debug enabled on TVM. "
+ "Profiling may also have an impact on inference time, "
+ "making it take longer to be generated.",
+ )
+ parser.add_argument(
+ "--repeat", metavar="N", type=int, default=1, help="repeat the run n times. Defaults to '1'"
+ )
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs_file=args.inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ .. note::
+ We can't simply get the input tensors from a TVM graph
+ because weight tensors are treated equivalently. Therefore, to
+ find the input tensors we look at the 'arg_nodes' in the graph
+ (which are either weights or inputs) and check which ones don't
+ appear in the params (where the weights are stored). These nodes
+ are therefore inferred to be input tensors.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+
+ shape_dict = {}
+ dtype_dict = {}
+ # Use a special function to load the binary params back into a dict
+ load_arr = tvm.get_global_func("tvm.relay._load_param_dict")(params)
+ param_names = [v.name for v in load_arr]
+ graph = json.loads(graph_str)
+ for node_id in graph["arg_nodes"]:
+ node = graph["nodes"][node_id]
+ # If a node is not in the params, infer it to be an input node
+ name = node["name"]
+ if name not in param_names:
+ shape_dict[name] = graph["attrs"]["shape"][1][node_id]
+ dtype_dict[name] = graph["attrs"]["dltype"][1][node_id]
+
+ logger.debug("collecting graph input shape and type:")
+ logger.debug("graph input shape: %s", shape_dict)
+ logger.debug("graph input type: %s", dtype_dict)
+
+ return shape_dict, dtype_dict
+
+
+def generate_tensor_data(shape, dtype, fill_mode):
+ """Generate data to produce a tensor of given shape and dtype.
+
+ Random data generation depends on the dtype. For int8 types,
+ random integers in the range 0->255 are generated. For all other
+ types, random floats are generated in the range -1->1 and then
+ cast to the appropriate dtype.
+
+ This is used to quickly generate some data to input the models, as
+ a way to check that compiled module is sane for running.
+
+ Parameters
+ ----------
+ shape : tuple
+ The shape of the tensor.
+ dtype : str
+ The dtype of the tensor.
+ fill_mode : str
+ The fill-mode to use, either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ tensor : np.array
+ The generated tensor as a np.array.
+ """
+ if fill_mode == "zeros":
+ tensor = np.zeros(shape=shape, dtype=dtype)
+ elif fill_mode == "ones":
+ tensor = np.ones(shape=shape, dtype=dtype)
+ elif fill_mode == "random":
+ if "int8" in dtype:
+ tensor = np.random.randint(128, size=shape, dtype=dtype)
+ else:
+ tensor = np.random.uniform(-1, 1, size=shape).astype(dtype)
+ else:
+ raise TVMCException("unknown fill-mode: {}".format(fill_mode))
+
+ return tensor
+
+
+def make_inputs_dict(inputs_file, shape_dict, dtype_dict, fill_mode):
+ """Make the inputs dictionary for a graph.
+
+ Use data from 'inputs' where specified. For input tensors
+ where no data has been given, generate data according to the
+ chosen fill-mode.
+
+ Parameters
+ ----------
+ inputs_file : str
+ Path to a .npz file containing the inputs.
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ fill_mode : str
+ The fill-mode to use when generating tensor data.
+ Can be either "zeros", "ones" or "random".
+
+ Returns
+ -------
+ inputs_dict : dict
+ Complete inputs dictionary - {input_name: np.array}.
+ """
+ logger.debug("creating inputs dict")
+
+ try:
+ inputs = np.load(inputs_file) if inputs_file else {}
+ except IOError as ex:
+ raise TVMCException("Error loading inputs file: %s" % ex)
+
+ # First check all the keys in inputs exist in the graph
+ for input_name in inputs:
+ if input_name not in shape_dict.keys():
+ raise TVMCException(
+ "the input tensor '{}' is not in the graph. Expected inputs: '{}'".format(
+ input_name, shape_dict.keys()
+ )
+ )
+
+ # Now construct the input dict, generating tensors where no
+ # data already exists in 'inputs'
+ inputs_dict = {}
+ for input_name in shape_dict:
+ if input_name in inputs.keys():
+ logger.debug("setting input '%s' with user input data", input_name)
+ inputs_dict[input_name] = inputs[input_name]
+ else:
+ shape = shape_dict[input_name]
+ dtype = dtype_dict[input_name]
+
+ logger.debug(
+ "generating data for input '%s' (shape: %s, dtype: %s), using fill-mode '%s'",
+ input_name,
+ shape,
+ dtype,
+ fill_mode,
+ )
+ data = generate_tensor_data(shape, dtype, fill_mode)
+ inputs_dict[input_name] = data
+
+ return inputs_dict
+
+
+def run_module(
+ module_file,
+ hostname,
+ port=9090,
+ rpc_key=None,
+ device=None,
+ inputs_file=None,
+ fill_mode="random",
+ repeat=1,
+ profile=False,
+):
+ """Run a compiled graph runtime module locally or remotely with
+ optional input values.
+
+ If input tensors are not specified explicitly, they can be filled
+ with zeroes, ones or random data.
+
+ Parameters
+ ----------
+ module_file : str
+ The path to the module file (a .tar file).
+ hostname : str
+ The hostname of the target device on which to run.
+ port : int, optional
+ The port of the target device on which to run.
+ rpc_key : str, optional
+ The tracker key of the target device. If this is set, it
+ will be assumed that remote points to a tracker.
+ device: str, optional
+ the device (e.g. "cpu" or "gpu") to be targeted by the RPC
+ session, local or remote).
+ inputs_file : str, optional
+ Path to an .npz file containing the inputs.
+ fill_mode : str, optional
+ The fill-mode to use when generating data for input tensors.
+ Valid options are "zeros", "ones" and "random".
+ Defaults to "zeros".
Review comment:
I'll fix that in a sec.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r498383091
##########
File path: tests/python/driver/tvmc/test_runner.py
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import pytest
+import numpy as np
+
+from tvm.driver import tvmc
+
+
+def test_generate_tensor_data_zeros():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "zeros")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data_ones():
+ expected_shape = (224, 224)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "ones")
+
+ assert sut.shape == (224, 224)
+
+
+def test_generate_tensor_data_random():
+ expected_shape = (2, 3)
+ expected_dtype = "uint8"
+ sut = tvmc.runner.generate_tensor_data(expected_shape, expected_dtype, "random")
+
+ assert sut.shape == (2, 3)
+
+
+def test_generate_tensor_data__type_unknown():
+ with pytest.raises(tvmc.common.TVMCException) as e:
+ tvmc.runner.generate_tensor_data((2, 3), "float32", "whatever")
+
+
+def test_format_times__contains_header():
+ sut = tvmc.runner.format_times([60, 120, 12, 42])
+ assert "std (s)" in sut
+
+
+def test_get_top_results_keep_results():
+ fake_outputs = {"output_0": np.array([[1, 2, 3, 4], [5, 6, 7, 8]])}
+ number_of_results_wanted = 3
+ sut = tvmc.runner.get_top_results(fake_outputs, number_of_results_wanted)
+
+ expected_number_of_lines = 2
+ assert len(sut) == expected_number_of_lines
+
+ expected_number_of_results_per_line = 3
+ assert len(sut[0]) == expected_number_of_results_per_line
+ assert len(sut[1]) == expected_number_of_results_per_line
+
+
+def test_run_tflite_module__with_profile__valid_input(
Review comment:
This is used on other test cases, mostly to mark separation of sentences. I can revert it back in case it looks too wrong, but for me they are very useful when reading it in a long list of tests with similar names and slightly different conditions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496753128
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
+ )
+ parser.add_argument("--repeat", metavar="N", type=int, default=1, help="repeat the run n times")
+ parser.add_argument(
+ "--rpc-key",
+ nargs=1,
+ help="the RPC tracker key of the target device",
+ )
+ parser.add_argument(
+ "--rpc-tracker",
+ nargs=1,
+ help="hostname (required) and port (optional, defaults to 9090) of the RPC tracker, "
+ "e.g. '192.168.0.100:9999'",
+ )
+ parser.add_argument("FILE", help="path to the compiled module file")
+
+
+def drive_run(args):
+ """Invoke runner module with command line arguments
+
+ Parameters
+ ----------
+ args: argparse.Namespace
+ Arguments from command line parser.
+ """
+ inputs = {}
+ if args.inputs:
+ inputs = np.load(args.inputs)
+
+ rpc_hostname, rpc_port = common.tracker_host_port_from_cli(args.rpc_tracker)
+
+ outputs, times = run_module(
+ args.FILE,
+ rpc_hostname,
+ rpc_port,
+ args.rpc_key,
+ inputs=inputs,
+ device=args.device,
+ fill_mode=args.fill_mode,
+ repeat=args.repeat,
+ profile=args.profile,
+ )
+
+ if args.print_time:
+ stat_table = format_times(times)
+ # print here is intentional
+ print(stat_table)
+
+ if args.print_top:
+ top_results = get_top_results(outputs, args.print_top)
+ # print here is intentional
+ print(top_results)
+
+ if args.outputs:
+ # Save the outputs
+ np.savez(args.outputs, **outputs)
+
+
+def get_input_info(graph_str, params):
+ """Return the 'shape' and 'dtype' dictionaries for the input
+ tensors of a compiled module.
+
+ Parameters
+ ----------
+ graph_str : str
+ JSON graph of the module serialized as a string.
+ params : bytearray
+ Params serialized as a bytearray.
+
+ Returns
+ -------
+ shape_dict : dict
+ Shape dictionary - {input_name: tuple}.
+ dtype_dict : dict
+ dtype dictionary - {input_name: dtype}.
+ """
+ # NOTE - We can't simply get the input tensors from a TVM graph
+ # because weight tensors are treated equivalently. Therefore, to
+ # find the input tensors we look at the 'arg_nodes' in the graph
+ # (which are either weights or inputs) and check which ones don't
+ # appear in the params (where the weights are stored). These nodes
+ # are therefore inferred to be input tensors.
Review comment:
Very interesting! I didn't know `.. note::` before. Done now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] leandron commented on a change in pull request #6578: [tvmc] Introduce 'run' subcommand (part 4/4)
Posted by GitBox <gi...@apache.org>.
leandron commented on a change in pull request #6578:
URL: https://github.com/apache/incubator-tvm/pull/6578#discussion_r496827216
##########
File path: python/tvm/driver/tvmc/runner.py
##########
@@ -0,0 +1,450 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Provides support to run compiled networks both locally and remotely.
+"""
+import json
+import logging
+import os
+import tarfile
+import tempfile
+
+import numpy as np
+import tvm
+from tvm import rpc
+from tvm.autotvm.measure import request_remote
+from tvm.contrib import graph_runtime as runtime
+from tvm.contrib.debugger import debug_runtime
+
+from . import common
+from .common import TVMCException
+from .main import register_parser
+
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("TVMC")
+
+
+@register_parser
+def add_run_parser(subparsers):
+ """ Include parser for 'run' subcommand """
+
+ parser = subparsers.add_parser("run", help="run a compiled module")
+ parser.set_defaults(func=drive_run)
+
+ # TODO --device needs to be extended and tested to support other targets,
+ # like 'cl', 'webgpu', etc (@leandron)
+ parser.add_argument(
+ "--device",
+ choices=["cpu", "gpu"],
+ default="cpu",
+ help="target device to run the compiled module",
+ )
+ parser.add_argument(
+ "--fill-mode",
+ choices=["zeros", "ones", "random"],
+ default="zeros",
+ help="fill all input tensors with values",
+ )
+ parser.add_argument("-i", "--inputs", help="path to the .npz input file")
+ parser.add_argument("-o", "--outputs", help="path to the .npz output file")
+ parser.add_argument(
+ "--print-time", action="store_true", help="record and print the execution time(s)"
+ )
+ parser.add_argument(
+ "--print-top",
+ metavar="N",
+ type=int,
+ help="print the top n values and indices of the output tensor",
+ )
+ parser.add_argument(
+ "--profile", action="store_true", help="generate profiling data from the runtime execution"
Review comment:
I updated the message now. Let me know whether is conveys the message you had in mind.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org