You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/09/27 13:32:08 UTC
[GitHub] [tvm] yelite opened a new pull request, #12914: Add a script to tune and benchmark models from TorchBench
yelite opened a new pull request, #12914:
URL: https://github.com/apache/tvm/pull/12914
This PR adds a script to tune and benchmark TorchBench models, using torchdynamo and the pytorch importer in TVM.
cc @junrushao @zxybazh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
junrushao commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981436091
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
Review Comment:
```suggestion
TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
junrushao commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981446701
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO: Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.INFO)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ return result
+
+ return forward
+
+
+def get_vm_forward(vm):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [tvm.nd.from_dlpack(arg.contiguous()) for arg in args]
+ result = vm.invoke("main", *args)
+ if IS_CUDA:
+ torch.cuda.synchronize()
+
+ if isinstance(result, tvm.nd.NDArray):
+ result = [result]
+ return [torch.from_dlpack(m) for m in result]
+
+ return forward
+
+
+def create_tvm_task_collection_backend(tasks):
+ """
+ This torchdynamo backend only collects the extracted tasks from Metaschedule.
+ It doesn't tune the model.
+ """
+
+ def backend(graph_module, example_inputs):
+ jit_mod = torch.jit.trace(graph_module, example_inputs)
+ shape_list = [(f"inp_{idx}", i.shape) for idx, i in enumerate(example_inputs)]
+ ir_mod, params = tvm.relay.frontend.from_pytorch(jit_mod, shape_list)
+
+ extracted_tasks = ms.extract_task_from_relay(ir_mod, ARGS.target, params)
+ logger.info("Extracted %d tasks", len(extracted_tasks))
+ tasks.extend(extracted_tasks)
+
+ return graph_module.forward
+
+ return backend
+
+
+def create_tvm_compilation_backend(database):
+ """
+ This torchdynamo backend compiles the model using history best record from the
+ Metaschedule database.
+ """
+
+ def backend(graph_module, example_inputs):
+ from tvm import relay
+ from tvm.ir.transform import PassContext
Review Comment:
should surround this with:
```python
# pylint: disable=import-outside-toplevel
# pylint: enable=import-outside-toplevel
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
junrushao commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981444609
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO: Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.INFO)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
Review Comment:
qq: is it going to incur an unnecessary extra copy if `arg` has been contiguous already, or it's going to be a no-op?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] yelite commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
yelite commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981713605
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO: Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.INFO)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
Review Comment:
My thoughts are:
1. We should call torch's synchronize before the computation here then call tvm's synchronize afterward, letting each library's sync to wait the computation from their side.
2. We probably want to move sync before `get_output`. I believe that it doesn't matter as long as `torch.from_dlpack(mod.get_output(i))` is zero-copy. But it doesn't hurt to move sync before that line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zxybazh commented on a diff in pull request #12914: [MetaSchedule] Add Script for TorchBench Model Tuning & Benchmarking
Posted by GitBox <gi...@apache.org>.
zxybazh commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r982840169
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,591 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+All available target tags (like nvidia/geforce-rtx-3070) can be found at
+https://github.com/apache/tvm/blob/main/src/target/tag.cc
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning (the database json files need to be inside
+of the work directory).
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode all \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+import warnings
+from enum import Enum
+from typing import Callable, List, Tuple
+
+import numpy as np # type: ignore
+import torch # type: ignore
+from scipy.stats import ttest_ind # type: ignore
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib.graph_executor import GraphModule
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+class RunMode(Enum):
+ """
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """
+
+ ALL = "all"
+ TUNE = "tune"
+ EVAL = "eval"
+
+ @property
+ def should_tune(self):
+ """
+ Returns whether it should tune the model.
+ """
+ return self != RunMode.EVAL
+
+ @property
+ def should_eval(self):
+ """
+ Returns whether it should actually benchmark the model.
+ """
+ return self != RunMode.TUNE
+
+
+class ResultComparisonMetric(Enum):
+ """
+ This changes how it compares the resultl with the expected value during
+ accuracy check.
+ - cosine: Use the cosine similarity. It should be greater than 0.99.
+ - allclose-1e-4: Use the max element-wise absolute difference. It should be less than 1e-4.
+ """
+
+ COSINE = "cosine"
+ ALLCLOSE = "allclose-1e-4"
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=RunMode,
+ default=RunMode.ALL,
+ help=RunMode.__doc__,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--result-metric",
+ type=ResultComparisonMetric,
+ default=ResultComparisonMetric.ALLCLOSE,
+ help=ResultComparisonMetric.__doc__,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+ args.add_argument(
+ "--benchmark-warmup-rounds",
+ type=int,
+ default=5,
+ help="The number of rounds to warmup before starting to measure the performance.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results and store databases for compilation.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(
+ IS_CUDA, cosine_similarity=ARGS.result_comparison_metric == ResultComparisonMetric.COSINE
+)
+import torchdynamo # type: ignore
Review Comment:
May I ask why is this import down here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
junrushao commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981440478
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO: Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.INFO)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
Review Comment:
add a warning indicating fall-back to local runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zxybazh commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
zxybazh commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r982661863
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -131,19 +155,23 @@ def parse_args():
help="The batch size of model input. Use TorchBench's default value if not specified.",
)
args.add_argument(
- "--cosine-similarity",
- action="store_true",
- help="""
- Whether to use cosine similarity to determine whether the output is the same as
- expected. By default torch.allclose is used.
- """,
+ "--result-comparison-metric",
Review Comment:
Nit, this seems quite long, what about `--result-metric`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
junrushao commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981438471
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
Review Comment:
add a link to TVM tags
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zxybazh commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
zxybazh commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981496373
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
Review Comment:
Ditto.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
Review Comment:
Just curious, can we customize other benchmarking details like warm up rounds, time between measurements, etc.?
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
Review Comment:
This should be based on whether target is cpu, no need to enable for cuda. Therefore, I suggest that if we use `store_true` we should throw a warning if this is not running on cpu.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
Review Comment:
This is ok but unnecessary.
##########
python/tvm/meta_schedule/testing/torchbench/utils.py:
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import os
+import sys
+from dataclasses import dataclass
+
+import torch
+
+
+def find_torchdynamo():
+ candidates = [
+ "torchdynamo",
+ "../torchdynamo",
+ "../../torchdynamo",
+ ]
+ for library_dir in candidates:
+ if os.path.exists(f"{library_dir}/benchmarks"):
+ return library_dir
+ else:
+ raise RuntimeError(
+ f"Cannot find directory for torchdynamo. You need to clone https://github.com/pytorch/torchdynamo to the parent directory of cwd."
+ )
+
+
+DYNAMO_DIR = find_torchdynamo()
+sys.path.append(DYNAMO_DIR)
+sys.path.append(f"{DYNAMO_DIR}/benchmarks")
+
+from benchmarks.common import same, timed
+from torchbench import TorchBenchmarkRunner
+
+
+def load_torchdynamo_benchmark_runner(is_cuda: bool, cosine_similarity: bool = False):
+ @dataclass
+ class RunnerArgs:
+ ci: bool = False
+ training: bool = False
+ use_eval_mode: bool = True
+ dynamic_shapes: bool = False
+ float16: bool = False
+ float32: bool = False
+
+ accuracy: bool = False
+ performance: bool = True
+
+ cosine: bool = False
+
+ args = RunnerArgs(cosine=cosine_similarity)
+
+ runner = TorchBenchmarkRunner()
+ runner.args = args
+ runner.model_iter_fn = runner.forward_pass
+
+ if is_cuda:
+ import benchmarks.common
Review Comment:
import-outside-toplevel
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ return result
+
+ return forward
+
+
+def get_vm_forward(vm):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [tvm.nd.from_dlpack(arg.contiguous()) for arg in args]
+ result = vm.invoke("main", *args)
+ if IS_CUDA:
+ torch.cuda.synchronize()
+
+ if isinstance(result, tvm.nd.NDArray):
+ result = [result]
+ return [torch.from_dlpack(m) for m in result]
+
+ return forward
+
+
+def create_tvm_task_collection_backend(tasks):
+ """
+ This torchdynamo backend only collects the extracted tasks from Metaschedule.
+ It doesn't tune the model.
+ """
+
+ def backend(graph_module, example_inputs):
+ jit_mod = torch.jit.trace(graph_module, example_inputs)
+ shape_list = [(f"inp_{idx}", i.shape) for idx, i in enumerate(example_inputs)]
+ ir_mod, params = tvm.relay.frontend.from_pytorch(jit_mod, shape_list)
+
+ extracted_tasks = ms.extract_task_from_relay(ir_mod, ARGS.target, params)
+ logger.info("Extracted %d tasks", len(extracted_tasks))
+ tasks.extend(extracted_tasks)
+
+ return graph_module.forward
+
+ return backend
+
+
+def create_tvm_compilation_backend(database):
+ """
+ This torchdynamo backend compiles the model using history best record from the
+ Metaschedule database.
+ """
+
+ def backend(graph_module, example_inputs):
Review Comment:
Type annotation.
##########
python/tvm/meta_schedule/testing/torchbench/utils.py:
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import os
+import sys
+from dataclasses import dataclass
+
+import torch
+
+
+def find_torchdynamo():
+ candidates = [
+ "torchdynamo",
+ "../torchdynamo",
+ "../../torchdynamo",
+ ]
+ for library_dir in candidates:
+ if os.path.exists(f"{library_dir}/benchmarks"):
+ return library_dir
+ else:
+ raise RuntimeError(
+ f"Cannot find directory for torchdynamo. You need to clone https://github.com/pytorch/torchdynamo to the parent directory of cwd."
+ )
+
+
+DYNAMO_DIR = find_torchdynamo()
+sys.path.append(DYNAMO_DIR)
+sys.path.append(f"{DYNAMO_DIR}/benchmarks")
+
+from benchmarks.common import same, timed
+from torchbench import TorchBenchmarkRunner
+
+
+def load_torchdynamo_benchmark_runner(is_cuda: bool, cosine_similarity: bool = False):
Review Comment:
Return type annotation
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
Review Comment:
You may want to add that the database json files need to be inside of the work directory.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
Review Comment:
...and store databases for compilation.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
Review Comment:
You may refer to `tune_relay.py` script for details for layout specification.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ return result
+
+ return forward
+
+
+def get_vm_forward(vm):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [tvm.nd.from_dlpack(arg.contiguous()) for arg in args]
+ result = vm.invoke("main", *args)
+ if IS_CUDA:
+ torch.cuda.synchronize()
+
+ if isinstance(result, tvm.nd.NDArray):
+ result = [result]
+ return [torch.from_dlpack(m) for m in result]
+
+ return forward
+
+
+def create_tvm_task_collection_backend(tasks):
Review Comment:
Type annotation.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ return result
+
+ return forward
+
+
+def get_vm_forward(vm):
Review Comment:
Type annotation.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
Review Comment:
Type annotation.
##########
python/tvm/meta_schedule/testing/torchbench/utils.py:
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import os
+import sys
+from dataclasses import dataclass
+
+import torch
+
+
+def find_torchdynamo():
Review Comment:
Type annotation.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ return result
+
+ return forward
+
+
+def get_vm_forward(vm):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [tvm.nd.from_dlpack(arg.contiguous()) for arg in args]
+ result = vm.invoke("main", *args)
+ if IS_CUDA:
+ torch.cuda.synchronize()
+
+ if isinstance(result, tvm.nd.NDArray):
+ result = [result]
+ return [torch.from_dlpack(m) for m in result]
+
+ return forward
+
+
+def create_tvm_task_collection_backend(tasks):
+ """
+ This torchdynamo backend only collects the extracted tasks from Metaschedule.
+ It doesn't tune the model.
+ """
+
+ def backend(graph_module, example_inputs):
Review Comment:
Type annotation.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ return result
+
+ return forward
+
+
+def get_vm_forward(vm):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [tvm.nd.from_dlpack(arg.contiguous()) for arg in args]
+ result = vm.invoke("main", *args)
+ if IS_CUDA:
+ torch.cuda.synchronize()
+
+ if isinstance(result, tvm.nd.NDArray):
+ result = [result]
+ return [torch.from_dlpack(m) for m in result]
+
+ return forward
+
+
+def create_tvm_task_collection_backend(tasks):
+ """
+ This torchdynamo backend only collects the extracted tasks from Metaschedule.
+ It doesn't tune the model.
+ """
+
+ def backend(graph_module, example_inputs):
+ jit_mod = torch.jit.trace(graph_module, example_inputs)
+ shape_list = [(f"inp_{idx}", i.shape) for idx, i in enumerate(example_inputs)]
+ ir_mod, params = tvm.relay.frontend.from_pytorch(jit_mod, shape_list)
+
+ extracted_tasks = ms.extract_task_from_relay(ir_mod, ARGS.target, params)
+ logger.info("Extracted %d tasks", len(extracted_tasks))
+ tasks.extend(extracted_tasks)
+
+ return graph_module.forward
+
+ return backend
+
+
+def create_tvm_compilation_backend(database):
+ """
+ This torchdynamo backend compiles the model using history best record from the
+ Metaschedule database.
+ """
+
+ def backend(graph_module, example_inputs):
+ # pylint: disable=import-outside-toplevel
+ from tvm import relay
+ from tvm.ir.transform import PassContext
+
+ # pylint: enable=import-outside-toplevel
+
+ jit_mod = torch.jit.trace(graph_module, example_inputs)
+ shape_list = [(f"inp_{idx}", i.shape) for idx, i in enumerate(example_inputs)]
+ ir_mod, params = tvm.relay.frontend.from_pytorch(jit_mod, shape_list)
+
+ relay_build = {"graph": relay.build, "vm": relay.vm.compile}[ARGS.backend]
+ with ARGS.target, ms.utils.autotvm_silencer(), database:
+ with PassContext(
+ opt_level=3,
+ config={
+ "relay.backend.use_meta_schedule": True,
+ "relay.backend.use_meta_schedule_dispatch": not IS_CUDA,
+ "relay.backend.tir_converter": "default",
+ },
+ ):
+ lib = relay_build(ir_mod, target=ARGS.target, params=params)
+
+ device = tvm.cuda(0) if IS_CUDA else tvm.cpu(0)
+
+ if ARGS.backend == "graph":
+ mod = graph_executor.GraphModule(lib["default"](device))
+ return get_graph_executor_forward(mod)
+ elif ARGS.backend == "vm":
+ vm = VirtualMachine(lib, device)
+ return get_vm_forward(vm)
+ else:
+ raise RuntimeError(f"Unknown backend {ARGS.backend}")
+
+ return backend
+
+
+def format_time(seconds):
+ if seconds > 1:
+ return f"{seconds:.3g}s"
+ else:
+ return f"{seconds * 1000:.3g}ms"
+
+
+def null_experiment(*args, **kwargs):
+ return ""
+
+
+def performance_experiment(model_iter_fn, model, example_inputs):
Review Comment:
Type annotation.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
Review Comment:
You can refer to the permalink of this file https://github.com/apache/tvm/blob/main/src/target/tag.cc
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ return result
+
+ return forward
+
+
+def get_vm_forward(vm):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [tvm.nd.from_dlpack(arg.contiguous()) for arg in args]
+ result = vm.invoke("main", *args)
+ if IS_CUDA:
+ torch.cuda.synchronize()
+
+ if isinstance(result, tvm.nd.NDArray):
+ result = [result]
+ return [torch.from_dlpack(m) for m in result]
+
+ return forward
+
+
+def create_tvm_task_collection_backend(tasks):
+ """
+ This torchdynamo backend only collects the extracted tasks from Metaschedule.
+ It doesn't tune the model.
+ """
+
+ def backend(graph_module, example_inputs):
+ jit_mod = torch.jit.trace(graph_module, example_inputs)
+ shape_list = [(f"inp_{idx}", i.shape) for idx, i in enumerate(example_inputs)]
+ ir_mod, params = tvm.relay.frontend.from_pytorch(jit_mod, shape_list)
+
+ extracted_tasks = ms.extract_task_from_relay(ir_mod, ARGS.target, params)
+ logger.info("Extracted %d tasks", len(extracted_tasks))
+ tasks.extend(extracted_tasks)
+
+ return graph_module.forward
+
+ return backend
+
+
+def create_tvm_compilation_backend(database):
Review Comment:
Type annotation.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ return result
+
+ return forward
+
+
+def get_vm_forward(vm):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [tvm.nd.from_dlpack(arg.contiguous()) for arg in args]
+ result = vm.invoke("main", *args)
+ if IS_CUDA:
+ torch.cuda.synchronize()
+
+ if isinstance(result, tvm.nd.NDArray):
+ result = [result]
+ return [torch.from_dlpack(m) for m in result]
+
+ return forward
+
+
+def create_tvm_task_collection_backend(tasks):
+ """
+ This torchdynamo backend only collects the extracted tasks from Metaschedule.
+ It doesn't tune the model.
+ """
+
+ def backend(graph_module, example_inputs):
+ jit_mod = torch.jit.trace(graph_module, example_inputs)
+ shape_list = [(f"inp_{idx}", i.shape) for idx, i in enumerate(example_inputs)]
+ ir_mod, params = tvm.relay.frontend.from_pytorch(jit_mod, shape_list)
+
+ extracted_tasks = ms.extract_task_from_relay(ir_mod, ARGS.target, params)
+ logger.info("Extracted %d tasks", len(extracted_tasks))
+ tasks.extend(extracted_tasks)
+
+ return graph_module.forward
+
+ return backend
+
+
+def create_tvm_compilation_backend(database):
+ """
+ This torchdynamo backend compiles the model using history best record from the
+ Metaschedule database.
+ """
+
+ def backend(graph_module, example_inputs):
+ # pylint: disable=import-outside-toplevel
+ from tvm import relay
+ from tvm.ir.transform import PassContext
+
+ # pylint: enable=import-outside-toplevel
+
+ jit_mod = torch.jit.trace(graph_module, example_inputs)
+ shape_list = [(f"inp_{idx}", i.shape) for idx, i in enumerate(example_inputs)]
+ ir_mod, params = tvm.relay.frontend.from_pytorch(jit_mod, shape_list)
+
+ relay_build = {"graph": relay.build, "vm": relay.vm.compile}[ARGS.backend]
+ with ARGS.target, ms.utils.autotvm_silencer(), database:
+ with PassContext(
+ opt_level=3,
+ config={
+ "relay.backend.use_meta_schedule": True,
+ "relay.backend.use_meta_schedule_dispatch": not IS_CUDA,
+ "relay.backend.tir_converter": "default",
+ },
+ ):
+ lib = relay_build(ir_mod, target=ARGS.target, params=params)
+
+ device = tvm.cuda(0) if IS_CUDA else tvm.cpu(0)
+
+ if ARGS.backend == "graph":
+ mod = graph_executor.GraphModule(lib["default"](device))
+ return get_graph_executor_forward(mod)
+ elif ARGS.backend == "vm":
+ vm = VirtualMachine(lib, device)
+ return get_vm_forward(vm)
+ else:
+ raise RuntimeError(f"Unknown backend {ARGS.backend}")
+
+ return backend
+
+
+def format_time(seconds):
Review Comment:
Type annotation.
##########
python/tvm/meta_schedule/testing/torchbench/utils.py:
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import os
+import sys
+from dataclasses import dataclass
+
+import torch
+
+
+def find_torchdynamo():
+ candidates = [
+ "torchdynamo",
+ "../torchdynamo",
+ "../../torchdynamo",
+ ]
+ for library_dir in candidates:
+ if os.path.exists(f"{library_dir}/benchmarks"):
+ return library_dir
+ else:
+ raise RuntimeError(
+ f"Cannot find directory for torchdynamo. You need to clone https://github.com/pytorch/torchdynamo to the parent directory of cwd."
+ )
+
+
+DYNAMO_DIR = find_torchdynamo()
+sys.path.append(DYNAMO_DIR)
+sys.path.append(f"{DYNAMO_DIR}/benchmarks")
+
+from benchmarks.common import same, timed
+from torchbench import TorchBenchmarkRunner
+
+
+def load_torchdynamo_benchmark_runner(is_cuda: bool, cosine_similarity: bool = False):
+ @dataclass
+ class RunnerArgs:
+ ci: bool = False
Review Comment:
A brief comment explaining these args are for torch dynamo runner configuration would be apprecidated.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ return result
+
+ return forward
+
+
+def get_vm_forward(vm):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [tvm.nd.from_dlpack(arg.contiguous()) for arg in args]
+ result = vm.invoke("main", *args)
+ if IS_CUDA:
+ torch.cuda.synchronize()
+
+ if isinstance(result, tvm.nd.NDArray):
+ result = [result]
+ return [torch.from_dlpack(m) for m in result]
+
+ return forward
+
+
+def create_tvm_task_collection_backend(tasks):
+ """
+ This torchdynamo backend only collects the extracted tasks from Metaschedule.
+ It doesn't tune the model.
+ """
+
+ def backend(graph_module, example_inputs):
+ jit_mod = torch.jit.trace(graph_module, example_inputs)
+ shape_list = [(f"inp_{idx}", i.shape) for idx, i in enumerate(example_inputs)]
+ ir_mod, params = tvm.relay.frontend.from_pytorch(jit_mod, shape_list)
+
+ extracted_tasks = ms.extract_task_from_relay(ir_mod, ARGS.target, params)
+ logger.info("Extracted %d tasks", len(extracted_tasks))
+ tasks.extend(extracted_tasks)
+
+ return graph_module.forward
+
+ return backend
+
+
+def create_tvm_compilation_backend(database):
+ """
+ This torchdynamo backend compiles the model using history best record from the
+ Metaschedule database.
+ """
+
+ def backend(graph_module, example_inputs):
+ # pylint: disable=import-outside-toplevel
+ from tvm import relay
+ from tvm.ir.transform import PassContext
+
+ # pylint: enable=import-outside-toplevel
+
+ jit_mod = torch.jit.trace(graph_module, example_inputs)
+ shape_list = [(f"inp_{idx}", i.shape) for idx, i in enumerate(example_inputs)]
+ ir_mod, params = tvm.relay.frontend.from_pytorch(jit_mod, shape_list)
+
+ relay_build = {"graph": relay.build, "vm": relay.vm.compile}[ARGS.backend]
+ with ARGS.target, ms.utils.autotvm_silencer(), database:
+ with PassContext(
+ opt_level=3,
+ config={
+ "relay.backend.use_meta_schedule": True,
+ "relay.backend.use_meta_schedule_dispatch": not IS_CUDA,
+ "relay.backend.tir_converter": "default",
+ },
+ ):
+ lib = relay_build(ir_mod, target=ARGS.target, params=params)
+
+ device = tvm.cuda(0) if IS_CUDA else tvm.cpu(0)
+
+ if ARGS.backend == "graph":
+ mod = graph_executor.GraphModule(lib["default"](device))
+ return get_graph_executor_forward(mod)
+ elif ARGS.backend == "vm":
+ vm = VirtualMachine(lib, device)
+ return get_vm_forward(vm)
+ else:
+ raise RuntimeError(f"Unknown backend {ARGS.backend}")
+
+ return backend
+
+
+def format_time(seconds):
+ if seconds > 1:
+ return f"{seconds:.3g}s"
+ else:
+ return f"{seconds * 1000:.3g}ms"
+
+
+def null_experiment(*args, **kwargs):
+ return ""
+
+
+def performance_experiment(model_iter_fn, model, example_inputs):
+ # Simplified from https://github.com/pytorch/torchdynamo/blob/c537639f9712621dc04ca09908796dbbe86c354b/benchmarks/common.py#L494
+ timings = np.zeros((ARGS.benchmark_repeat, 2), np.float64)
+
+ is_correct = True
+
+ frozen_model_iter_fn = torchdynamo.run(model_iter_fn)
+ for rep in range(ARGS.benchmark_repeat):
+ # interleave the runs to handle frequency scaling and load changes
+ timings[rep, 0], expected_output = timed(
+ model, model_iter_fn, example_inputs, return_result=True
+ )
+ timings[rep, 1], actual_output = timed(
+ model, frozen_model_iter_fn, example_inputs, return_result=True
+ )
+ is_correct = is_correct and same(
+ expected_output, actual_output, cos_similarity=ARGS.cosine_similarity
+ )
+
+ pvalue = ttest_ind(timings[:, 0], timings[:, 1]).pvalue
+ median = np.median(timings, axis=0)
+ speedup = median[0] / median[1]
+ logger.info(
+ f"eager:{format_time(median[0])} optimized:{format_time(median[1])} speedup:{speedup:.3f}x p:{pvalue:.3f}"
+ )
+ if not is_correct:
+ logger.error(f"Result is incorrect.")
+ logger.error(f"Expected (PyTorch eager): {expected_output}")
+ logger.error(f"Actual (Optimized): {actual_output}")
+
+ return ""
+
+
+def get_torch_device_type(target: tvm.target.Target) -> str:
+ if target.kind.name == "llvm":
+ return "cpu"
+ else:
+ return target.kind.name
Review Comment:
Since we may only test on cuda / llvm, maybe it's better to throw an exception when the given target is not these devices, e.g., hexagon?
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
Review Comment:
Maybe add a check for rpc_port & rpc_key as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao merged pull request #12914: [MetaSchedule] Add Script for TorchBench Model Tuning & Benchmarking
Posted by GitBox <gi...@apache.org>.
junrushao merged PR #12914:
URL: https://github.com/apache/tvm/pull/12914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
junrushao commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981448078
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO: Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.INFO)
Review Comment:
```suggestion
logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] yelite commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
yelite commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r982591784
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,511 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+All available target tags (like nvidia/geforce-rtx-3070) can be found at
+https://github.com/apache/tvm/blob/main/src/target/tag.cc
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
Review Comment:
This option is created to support running this script on a machine without the target GPU. For example, the tuning can be done with the help of RPC runners on a machine with 3070 while targeting A100.
We still require the host machine to have GPU because the model provided from TorchBench could potentially be different on CPU versus on CUDA. If we implement the remote task extraction (we probably will), we can even run this script on a machine without GPU to tune the model.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] yelite commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
yelite commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981713605
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO: Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.INFO)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
Review Comment:
My thoughts are:
1. We should call torch's synchronize before the computation here then call tvm's synchronize afterward, letting each library's sync to block computation from their side.
2. We probably want to move sync before `get_output`. I believe that it doesn't matter as long as `torch.from_dlpack(mod.get_output(i))` is zero-copy. But it doesn't hurt to move sync before that line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] yelite commented on a diff in pull request #12914: [MetaSchedule] Add Script for TorchBench Model Tuning & Benchmarking
Posted by GitBox <gi...@apache.org>.
yelite commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r982847452
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,591 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+All available target tags (like nvidia/geforce-rtx-3070) can be found at
+https://github.com/apache/tvm/blob/main/src/target/tag.cc
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning (the database json files need to be inside
+of the work directory).
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode all \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+import warnings
+from enum import Enum
+from typing import Callable, List, Tuple
+
+import numpy as np # type: ignore
Review Comment:
They are for suppressing errors like `Cannot find implementation or library stub for module named "torch"`(https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-12914/7/pipeline#step-97-log-74)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] yelite commented on a diff in pull request #12914: [MetaSchedule] Add Script for TorchBench Model Tuning & Benchmarking
Posted by GitBox <gi...@apache.org>.
yelite commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r982850617
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,591 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+All available target tags (like nvidia/geforce-rtx-3070) can be found at
+https://github.com/apache/tvm/blob/main/src/target/tag.cc
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning (the database json files need to be inside
+of the work directory).
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode all \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+import warnings
+from enum import Enum
+from typing import Callable, List, Tuple
+
+import numpy as np # type: ignore
+import torch # type: ignore
+from scipy.stats import ttest_ind # type: ignore
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib.graph_executor import GraphModule
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+class RunMode(Enum):
+ """
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """
+
+ ALL = "all"
+ TUNE = "tune"
+ EVAL = "eval"
+
+ @property
+ def should_tune(self):
+ """
+ Returns whether it should tune the model.
+ """
+ return self != RunMode.EVAL
+
+ @property
+ def should_eval(self):
+ """
+ Returns whether it should actually benchmark the model.
+ """
+ return self != RunMode.TUNE
+
+
+class ResultComparisonMetric(Enum):
+ """
+ This changes how it compares the resultl with the expected value during
+ accuracy check.
+ - cosine: Use the cosine similarity. It should be greater than 0.99.
+ - allclose-1e-4: Use the max element-wise absolute difference. It should be less than 1e-4.
+ """
+
+ COSINE = "cosine"
+ ALLCLOSE = "allclose-1e-4"
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=RunMode,
+ default=RunMode.ALL,
+ help=RunMode.__doc__,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--result-metric",
+ type=ResultComparisonMetric,
+ default=ResultComparisonMetric.ALLCLOSE,
+ help=ResultComparisonMetric.__doc__,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+ args.add_argument(
+ "--benchmark-warmup-rounds",
+ type=int,
+ default=5,
+ help="The number of rounds to warmup before starting to measure the performance.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results and store databases for compilation.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(
+ IS_CUDA, cosine_similarity=ARGS.result_comparison_metric == ResultComparisonMetric.COSINE
+)
+import torchdynamo # type: ignore
Review Comment:
It shouldn't be here. I moved it up. Thanks for catching this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] yelite commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
yelite commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981523847
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
Review Comment:
The earliest wheel on that index was built on 07/30. So I guess the retention is 60 days. Those wheels are pretty large in size (1 GB). I added a message in the comment string to suggest trying latest nightly if this one can not be found
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
junrushao commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981437773
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
Review Comment:
Quick quesiton: how long does it guarantee pytorch nightly to persist on the URL? I mean, is this instruction going to expire some time in the future because of server cleanup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
junrushao commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981439466
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
Review Comment:
generalize this option into `--metric`, whose default is, for example, `--allclose-1e-4`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] shingjan commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
shingjan commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981737528
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,511 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+All available target tags (like nvidia/geforce-rtx-3070) can be found at
+https://github.com/apache/tvm/blob/main/src/target/tag.cc
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
Review Comment:
NIT: As no perf evaluation will be done with `--tuning`, I feel like we should combine `tuning` and `all`, if perf evaluation doesn't really take much time.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,511 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+All available target tags (like nvidia/geforce-rtx-3070) can be found at
+https://github.com/apache/tvm/blob/main/src/target/tag.cc
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+import warnings
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ warnings.warn("Falling back to Metaschedule LocalRunner because --rpc-host isn't provided.")
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod, device):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
Review Comment:
i think this could potentially be a problem and that is reason why in torchdynamo's TVM backend torch.Tensor is converted to numpy and then to TVM.NDarray. And if the arg is typed `torch.Tensor` you may need `torch.utils.dlpack.to_dlpack(arg)` for this approach as well.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
Review Comment:
In torchdynamo there is warm up rounds. Adding this could be an option here.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
Review Comment:
I think the `--cosine-similarity` idea is from torchdynamo. I am okay with this argument as some models on CUDA will fail `allclose` anyway and we need to specify `--cosine` for those models.
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,511 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+All available target tags (like nvidia/geforce-rtx-3070) can be found at
+https://github.com/apache/tvm/blob/main/src/target/tag.cc
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
Review Comment:
maybe better to specify the `--mode all` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
junrushao commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981443837
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO: Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.INFO)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
+ )
+ mod.run()
+ result = [torch.from_dlpack(mod.get_output(i)) for i in range(mod.get_num_outputs())]
+ if IS_CUDA:
+ torch.cuda.synchronize()
Review Comment:
let's think twice about the synchronization here. two questions:
- `torch.cuda.synchronize` or `tvm.cuda(0).sync()`
- do we want to move the `sync` one line up before `mod.get_output()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] yelite commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
yelite commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r982585196
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,508 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
Review Comment:
Benchmark runner from TorchDynamo doesn't have these parameters exposed (Their args can be found at https://github.com/pytorch/torchdynamo/blob/main/benchmarks/common.py#L1363). But we can still implement these customization if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] yelite commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
yelite commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981525131
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO: Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.INFO)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
Review Comment:
It's no-op if arg is contiguous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] yelite commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
yelite commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r982611541
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,511 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+All available target tags (like nvidia/geforce-rtx-3070) can be found at
+https://github.com/apache/tvm/blob/main/src/target/tag.cc
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+import warnings
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ warnings.warn("Falling back to Metaschedule LocalRunner because --rpc-host isn't provided.")
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod, device):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
Review Comment:
That TVM backend actually doesn't work on CUDA. Converting tensor to numpy array will fail if it's on CUDA
```
>>> torch.zeros((5, 5), device="cuda").numpy()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
```
I guess the only problem here is from the boolean tensor (an maybe also tensor with unaligned memory). I will create a followup PR which uses Yaoda's work on TVM PyTorch integration to replace these code. That integration can handle these edge cases with minimal numbers of data copies.
`to_dlpack` is considered as a legacy interface (https://pytorch.org/docs/stable/dlpack.html#torch.utils.dlpack.to_dlpack). The new approach is to have a `__dlpack__` method on the object (like `torch.Tensor`) and the importing function (like `tvm.nd.from_dlpack`) can call it to get the capsule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
junrushao commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981439932
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
+# torchdynamo requires nightly pytorch
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO: Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
Review Comment:
```suggestion
# TODO(@yelite): Add a layout arg to transform the network after
# ingesting into Relay and before feeding into MetaSchedule.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
junrushao commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r981435678
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,504 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the torchdynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from torchdynamo.
+
+torchdynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in torchdynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```
Review Comment:
```suggestion
the appropriate Python environment:
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] yelite commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
yelite commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r982611541
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,511 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+All available target tags (like nvidia/geforce-rtx-3070) can be found at
+https://github.com/apache/tvm/blob/main/src/target/tag.cc
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning.
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+import warnings
+
+import numpy as np
+import torch
+from scipy.stats import ttest_ind
+
+import tvm
+import tvm.relay
+from tvm import meta_schedule as ms
+from tvm.contrib import graph_executor
+from tvm.meta_schedule.testing.torchbench.utils import (
+ load_torchdynamo_benchmark_runner,
+ same,
+ timed,
+)
+from tvm.runtime.vm import VirtualMachine
+from tvm.support import describe
+
+
+def parse_args():
+ args = argparse.ArgumentParser()
+
+ args.add_argument(
+ "--mode",
+ type=str,
+ choices=["tune", "eval", "all"],
+ default="all",
+ help="""
+ The running mode of this script. Available values are:
+ - tune: Only tune the model and create the tuning database.
+ - eval: Only benchmark model using pre-existing tuning database.
+ - all: Run both tuning and benchmark
+ """,
+ )
+ args.add_argument(
+ "--batch-size",
+ type=int,
+ default=None,
+ help="The batch size of model input. Use TorchBench's default value if not specified.",
+ )
+ args.add_argument(
+ "--cosine-similarity",
+ action="store_true",
+ help="""
+ Whether to use cosine similarity to determine whether the output is the same as
+ expected. By default torch.allclose is used.
+ """,
+ )
+ args.add_argument(
+ "--benchmark-repeat",
+ type=int,
+ default=10,
+ help="The number of times to repeat the benchmark measurement.",
+ )
+
+ # Model selection
+ args.add_argument(
+ "--model",
+ type=str,
+ required=True,
+ help="""
+ The name of model to run. It should a directory name under
+ https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models.
+ """,
+ )
+
+ # Tuning-related config
+ args.add_argument(
+ "--target",
+ type=tvm.target.Target,
+ required=True,
+ help="The target to tune and run benchmark for.",
+ )
+ args.add_argument(
+ "--work-dir",
+ type=str,
+ required=True,
+ help="The working directory to save intermediate results.",
+ )
+ args.add_argument(
+ "--cache-dir",
+ type=str,
+ default=None,
+ help="""
+ The directory to cache the generated network.
+ If not specified, the cache will be disabled.
+ """,
+ )
+ args.add_argument(
+ "--num-trials",
+ type=int,
+ required=True,
+ help="The max number of trials to run MetaSchedule.",
+ )
+ args.add_argument(
+ "--max-trials-per-task",
+ type=int,
+ default=None,
+ help="""
+ The max number of trials to run per task extracted in MetaSchedule.
+ By default it's the same as --num-trials.
+ """,
+ )
+ args.add_argument(
+ "--backend",
+ type=str,
+ choices=["graph", "vm"],
+ default="graph",
+ help="The backend to use for relay compilation(graph / vm).",
+ )
+ # TODO(@yelite): Add a layout arg to transform the network after
+ # ingesting into Relay and before feeding into MetaSchedule.
+
+ # Evaluator-related config
+ args.add_argument(
+ "--number",
+ type=int,
+ default=3,
+ help="The number of times to run the model for taking average in a single measurement.",
+ )
+ args.add_argument(
+ "--repeat",
+ type=int,
+ default=1,
+ help="The number of times to repeat the measurement.",
+ )
+ args.add_argument(
+ "--min-repeat-ms",
+ type=int,
+ default=100,
+ help="""
+ Minimum repeat time in ms. The number of runs will be increased if the actual
+ repeat time is lowered than this.
+ """,
+ )
+ args.add_argument(
+ "--adaptive-training",
+ action="store_true",
+ help="Whether to use adpative training for cost model.",
+ )
+ args.add_argument(
+ "--cpu-flush",
+ action="store_true",
+ help="Whether to perform CPU cache flush.",
+ )
+
+ # RPC-related args
+ args.add_argument(
+ "--rpc-host",
+ type=str,
+ help="Host of the RPC Tracker for tuning. Use LocalRunner if not provided",
+ )
+ args.add_argument(
+ "--rpc-port",
+ type=int,
+ help="Port of the RPC Tracker for tuning",
+ )
+ args.add_argument(
+ "--rpc-key",
+ type=str,
+ help="Key of the RPC Tracker for tuning",
+ )
+
+ parsed = args.parse_args()
+ return parsed
+
+
+logging.basicConfig(
+ format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+)
+logging.getLogger("tvm.meta_schedule").setLevel(logging.DEBUG)
+ARGS = parse_args()
+IS_CUDA = ARGS.target.kind.name == "cuda"
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+runner = load_torchdynamo_benchmark_runner(IS_CUDA)
+import torchdynamo
+
+
+def get_metaschedule_runner():
+ if ARGS.rpc_host is not None:
+ return ms.runner.RPCRunner(
+ rpc_config=ms.runner.RPCConfig(
+ tracker_host=ARGS.rpc_host,
+ tracker_port=ARGS.rpc_port,
+ tracker_key=ARGS.rpc_key,
+ session_timeout_sec=600,
+ ),
+ evaluator_config=ms.runner.EvaluatorConfig(
+ number=ARGS.number,
+ repeat=ARGS.repeat,
+ min_repeat_ms=ARGS.min_repeat_ms,
+ enable_cpu_cache_flush=ARGS.cpu_flush,
+ ),
+ alloc_repeat=1,
+ )
+ else:
+ warnings.warn("Falling back to Metaschedule LocalRunner because --rpc-host isn't provided.")
+ return ms.runner.LocalRunner()
+
+
+def get_tune_config():
+ if ARGS.mode == "eval":
+ max_trials_per_task = 0
+ max_trials_global = 0
+ else:
+ max_trials_per_task = ARGS.max_trials_per_task
+ max_trials_global = ARGS.num_trials
+
+ if max_trials_per_task is None:
+ max_trials_per_task = max_trials_global
+
+ return ms.TuneConfig(
+ strategy="evolutionary",
+ num_trials_per_iter=64,
+ max_trials_per_task=max_trials_per_task,
+ max_trials_global=max_trials_global,
+ adaptive_training=ARGS.adaptive_training,
+ )
+
+
+def get_graph_executor_forward(mod, device):
+ def forward(*args):
+ if IS_CUDA:
+ torch.cuda.synchronize()
+ args = [arg.contiguous() for arg in args]
+ for idx, arg in enumerate(args, 0):
+ mod.set_input(
+ f"inp_{idx}",
+ tvm.nd.from_dlpack(arg),
Review Comment:
That TVM backend actually doesn't work on CUDA. Converting tensor to numpy array will fail if it's on CUDA
```
>>> torch.zeros((5, 5), device="cuda").numpy()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
```
I guess the only problem here is from the boolean tensor (an maybe also tensor with unaligned memory). I will use Yaoda's work on TVM PyTorch integration to replace these code. That integration can handle these edge cases with minimal numbers of data copies.
`to_dlpack` is considered as a legacy interface (https://pytorch.org/docs/stable/dlpack.html#torch.utils.dlpack.to_dlpack). The new approach is to have a `__dlpack__` method on the object (like `torch.Tensor`) and the importing function (like `tvm.nd.from_dlpack`) can call it to get the capsule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zxybazh commented on a diff in pull request #12914: Add a script to tune and benchmark models from TorchBench
Posted by GitBox <gi...@apache.org>.
zxybazh commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r982661863
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -131,19 +155,23 @@ def parse_args():
help="The batch size of model input. Use TorchBench's default value if not specified.",
)
args.add_argument(
- "--cosine-similarity",
- action="store_true",
- help="""
- Whether to use cosine similarity to determine whether the output is the same as
- expected. By default torch.allclose is used.
- """,
+ "--result-comparison-metric",
Review Comment:
Nit, this seems quite long, what about `--result-metric` or `--compare-metric`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zxybazh commented on a diff in pull request #12914: [MetaSchedule] Add Script for TorchBench Model Tuning & Benchmarking
Posted by GitBox <gi...@apache.org>.
zxybazh commented on code in PR #12914:
URL: https://github.com/apache/tvm/pull/12914#discussion_r982838490
##########
python/tvm/meta_schedule/testing/torchbench/run.py:
##########
@@ -0,0 +1,591 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This script is for benchmarking TVM performance on models from TorchBench.
+It uses the TorchDynamo as the frontend to ingest models into TVM, and it also
+leverages the benchmark util from TorchDynamo.
+
+TorchDynamo (https://github.com/pytorch/torchdynamo) and TorchBench
+(https://github.com/pytorch/benchmark) need to be in the parent directory of TVM.
+We need a local clone of these repos because torchbench and the benchmark runner
+in TorchDynamo isn't designed to be used as a Python package.
+
+To setup the environment, run the following commands in the parent directory of TVM and with
+the appropriate Python environment:
+```bash
+# torchdynamo requires nightly pytorch. If it fails to find the specified version, try
+# installing the latest nightly pytorch.
+pip3 install --pre \
+ --extra-index-url https://download.pytorch.org/whl/nightly/cu116 \
+ torch==1.13.0.dev20220926 \
+ torchvision==0.14.0.dev20220926 \
+ torchtext==0.14.0.dev20220926
+
+git clone https://github.com/pytorch/torchdynamo
+pushd torchdynamo
+git checkout c537639f9712621dc04ca09908796dbbe86c354b
+pip install -e .
+popd
+
+sudo apt install git-lfs # git lfs is used for TorchBench
+git clone https://github.com/pytorch/benchmark
+pushd benchmark
+python install.py --continue_on_fail # fambench_xlmr might fail to install
+popd
+```
+
+To run a benchmark, the script can be run under 'tune' mode by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode tune \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 20000 \
+ --rpc-host <rpc tracker host for tuning> \
+ --rpc-port <rpc tracker port for tuning> \
+ --rpc-key <rpc key> \
+```
+
+All available target tags (like nvidia/geforce-rtx-3070) can be found at
+https://github.com/apache/tvm/blob/main/src/target/tag.cc
+
+Then the script can be run under 'eval' mode to actual benchmark the performance,
+using the tuning database under the work directory. This can be executed on a different
+machine than the one executes tuning (the database json files need to be inside
+of the work directory).
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode eval \
+ --model resnet50 \
+ --target "nvidia/geforce-rtx-3070" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+
+Alternatively, both tuning and evaluation can be done in a single run on the same machine,
+by
+```bash
+python python/tvm/meta_schedule/testing/torchbench/run.py \
+ --mode all \
+ --model resnet50 \
+ --target "llvm -num-cores 6" \
+ --work-dir ../workdir \
+ --num-trials 0
+```
+"""
+
+import argparse
+import functools
+import logging
+import warnings
+from enum import Enum
+from typing import Callable, List, Tuple
+
+import numpy as np # type: ignore
Review Comment:
Not sure if we need `type: ignore` here for imported libraries, any particular reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org