You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/06/21 07:52:01 UTC
[GitHub] [tvm] zxybazh opened a new pull request, #11797: [MetaSchedule] Tuning Script Upgrade
zxybazh opened a new pull request, #11797:
URL: https://github.com/apache/tvm/pull/11797
This PR introduces changes to upgrade the onnx / relay tuning script:
- [x] support int8 / uint8 as testing input for onnx and relay
- [x] refactor functions to clearly display per layer and e2e performance.
- [ ] support using VirtualMachine in addtion to GraphModule as a script option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zxybazh commented on pull request #11797: [MetaSchedule] Tuning Script Upgrade
Posted by GitBox <gi...@apache.org>.
zxybazh commented on PR #11797:
URL: https://github.com/apache/tvm/pull/11797#issuecomment-1169710770
Rebase and ready for review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 commented on a diff in pull request #11797: [MetaSchedule] Tuning Script Upgrade
Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on code in PR #11797:
URL: https://github.com/apache/tvm/pull/11797#discussion_r910457107
##########
python/tvm/meta_schedule/testing/utils.py:
##########
@@ -77,3 +81,145 @@ def apply_fixed_schedules(
database.commit_tuning_record(tune_rec)
return database
+
+
+def generate_input_data(input_shape: List[int], input_dtype: str) -> np.ndarray:
+ """Generate input date with given shape and data type.
+
+ Parameters
+ ----------
+ input_shape : List[int]
+ The shape of the input data.
+ input_dtype : str
+ The data type of the input date.
+
+ Returns
+ -------
+ input_data : np.ndarray
+ The generated input data with given shape and data type in numpy ndarray.
+ """
+ if input_dtype.startswith("float"):
+ return np.random.uniform(size=input_shape).astype(input_dtype)
+ if input_dtype in ["uint8", "int8"]:
+ return np.random.randint(
+ low=0,
+ high=127,
+ size=input_shape,
+ dtype="int32", # TODO(zxybazh): fix the datatype when int8 / uint8 is supported better
+ )
+ if input_dtype in ["int32", "int64"]:
+ return np.random.randint(low=0, high=10000, size=input_shape, dtype=input_dtype)
+ raise ValueError("Unsupported input datatype!")
+
+
+def f_timer(backend: str) -> Callable:
Review Comment:
let's find a better name. how about, `create_timer_func`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 commented on a diff in pull request #11797: [MetaSchedule] Tuning Script Upgrade
Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on code in PR #11797:
URL: https://github.com/apache/tvm/pull/11797#discussion_r910456672
##########
python/tvm/meta_schedule/testing/utils.py:
##########
@@ -77,3 +81,145 @@ def apply_fixed_schedules(
database.commit_tuning_record(tune_rec)
return database
+
+
+def generate_input_data(input_shape: List[int], input_dtype: str) -> np.ndarray:
+ """Generate input date with given shape and data type.
+
+ Parameters
+ ----------
+ input_shape : List[int]
+ The shape of the input data.
+ input_dtype : str
+ The data type of the input date.
+
+ Returns
+ -------
+ input_data : np.ndarray
+ The generated input data with given shape and data type in numpy ndarray.
+ """
+ if input_dtype.startswith("float"):
+ return np.random.uniform(size=input_shape).astype(input_dtype)
+ if input_dtype in ["uint8", "int8"]:
+ return np.random.randint(
+ low=0,
+ high=127,
+ size=input_shape,
+ dtype="int32", # TODO(zxybazh): fix the datatype when int8 / uint8 is supported better
+ )
+ if input_dtype in ["int32", "int64"]:
+ return np.random.randint(low=0, high=10000, size=input_shape, dtype=input_dtype)
Review Comment:
If it's an indexing table used in embedding lookups, then `low` and `high` here probably indicate the size of the embedding table. If not taken care of, could cause some buffer access overflow. I don't have good ideas of how to properly handling this, but let's add a warning here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zxybazh commented on a diff in pull request #11797: [MetaSchedule] Tuning Script Upgrade
Posted by GitBox <gi...@apache.org>.
zxybazh commented on code in PR #11797:
URL: https://github.com/apache/tvm/pull/11797#discussion_r910464495
##########
python/tvm/meta_schedule/testing/utils.py:
##########
@@ -77,3 +81,145 @@ def apply_fixed_schedules(
database.commit_tuning_record(tune_rec)
return database
+
+
+def generate_input_data(input_shape: List[int], input_dtype: str) -> np.ndarray:
+ """Generate input date with given shape and data type.
+
+ Parameters
+ ----------
+ input_shape : List[int]
+ The shape of the input data.
+ input_dtype : str
+ The data type of the input date.
+
+ Returns
+ -------
+ input_data : np.ndarray
+ The generated input data with given shape and data type in numpy ndarray.
+ """
+ if input_dtype.startswith("float"):
+ return np.random.uniform(size=input_shape).astype(input_dtype)
+ if input_dtype in ["uint8", "int8"]:
+ return np.random.randint(
+ low=0,
+ high=127,
+ size=input_shape,
+ dtype="int32", # TODO(zxybazh): fix the datatype when int8 / uint8 is supported better
+ )
+ if input_dtype in ["int32", "int64"]:
+ return np.random.randint(low=0, high=10000, size=input_shape, dtype=input_dtype)
Review Comment:
That's right, for `gpt-2` and `bert` the embedding table is not the same. In that case we need user input, so I'll just add the low and high as argument in this function and throw a warning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 commented on a diff in pull request #11797: [MetaSchedule] Tuning Script Upgrade
Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on code in PR #11797:
URL: https://github.com/apache/tvm/pull/11797#discussion_r910457244
##########
python/tvm/meta_schedule/testing/utils.py:
##########
@@ -77,3 +81,145 @@ def apply_fixed_schedules(
database.commit_tuning_record(tune_rec)
return database
+
+
+def generate_input_data(input_shape: List[int], input_dtype: str) -> np.ndarray:
+ """Generate input date with given shape and data type.
+
+ Parameters
+ ----------
+ input_shape : List[int]
+ The shape of the input data.
+ input_dtype : str
+ The data type of the input date.
+
+ Returns
+ -------
+ input_data : np.ndarray
+ The generated input data with given shape and data type in numpy ndarray.
+ """
+ if input_dtype.startswith("float"):
+ return np.random.uniform(size=input_shape).astype(input_dtype)
+ if input_dtype in ["uint8", "int8"]:
+ return np.random.randint(
+ low=0,
+ high=127,
+ size=input_shape,
+ dtype="int32", # TODO(zxybazh): fix the datatype when int8 / uint8 is supported better
+ )
+ if input_dtype in ["int32", "int64"]:
+ return np.random.randint(low=0, high=10000, size=input_shape, dtype=input_dtype)
+ raise ValueError("Unsupported input datatype!")
+
+
+def f_timer(backend: str) -> Callable:
+ """Create a function to run and benchmark the performance of whole given runtime module,
+ or Executable in relay vm.
+
+ Parameters
+ ----------
+ backend : str
+ The backend to use, graph / vm.
+
+ Returns
+ -------
+ func : Callable
+ The function to benchmark the workload.
+ """
+
+ def f_timer_func(
+ rt_mod: Union[tvm.runtime.Module, tvm.runtime.vm.Executable],
+ dev: tvm.device,
+ input_data: Dict[str, NDArray],
+ ) -> None:
+ """Run and benchmark the given runtime module, print out the result.
+
+ Parameters
+ ----------
+ rt_mod : Union[tvm.runtime.Module, tvm.runtime.vm.Executable]
+ The runtime module or vm executable.
+ dev : tvm.device
+ The device type to run workload.
+ input_data : Dict[str, np.ndarray]
+ The input data as a dictionary.
+ """
+ from tvm.contrib.graph_executor import GraphModule # pylint:disable=import-outside-toplevel
+ from tvm.runtime.vm import VirtualMachine # pylint:disable=import-outside-toplevel
+
+ if backend == "vm":
+ vm = VirtualMachine(rt_mod, dev) # pylint: disable=invalid-name
+ ftimer = vm.benchmark(
+ dev, min_repeat_ms=500, repeat=5, number=1, end_to_end=False, **input_data
+ )
+ elif backend == "graph":
+ mod = GraphModule(rt_mod["default"](dev))
+ for input_name, input_value in input_data.items():
+ mod.set_input(input_name, input_value)
+ ftimer = mod.module.time_evaluator("run", dev, min_repeat_ms=500, repeat=5, number=1)()
+ else:
+ raise ValueError(f"Backend {backend} not supported in f_timer!")
+
+ results = list(np.array(ftimer.results) * 1000.0) # type: ignore
+
+ print("Running time in time_evaluator: ", results)
+ print("-------------------------------")
+ print(f" Min (ms) : {min(results)}")
+ print(f" Max (ms) : {max(results)}")
+ print(f" Median (ms) : {median(results)}")
+ print(f"Average (ms) : {sum(results) / len(results)}")
+
+ return f_timer_func
+
+
+def f_per_layer(graph: str) -> Callable:
Review Comment:
ditto. how about `create_timer_per_layer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zxybazh commented on a diff in pull request #11797: [MetaSchedule] Tuning Script Upgrade
Posted by GitBox <gi...@apache.org>.
zxybazh commented on code in PR #11797:
URL: https://github.com/apache/tvm/pull/11797#discussion_r910461082
##########
python/tvm/meta_schedule/testing/custom_builder_runner.py:
##########
@@ -158,15 +160,22 @@ def run_module_via_rpc(
# pylint: enable=import-outside-toplevel
- with tempfile.TemporaryDirectory() as tmp_dir:
- filename = os.path.join(tmp_dir, "tvm_tmp_mod." + tar.output_format)
- lib.export_library(filename, tar)
- session = rpc_config.connect_server()
- session.upload(filename)
- _, filename = os.path.split(filename)
- rt_mod = session.load_module(filename)
- dev = session.device(dev_type=dev_type, dev_id=0)
- nd_args = {}
- for arg_key, arg_value in args.items():
- nd_args[arg_key] = ndarray.array(arg_value, dev)
- return continuation(rt_mod, dev, nd_args)
+ try:
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ filename = os.path.join(tmp_dir, "tvm_tmp_mod." + tar.output_format)
+ if backend == "vm":
+ code, lib = lib.save()
+ lib.export_library(filename, tar)
+ session = rpc_config.connect_server()
+ session.upload(filename)
+ _, filename = os.path.split(filename)
+ rt_mod = session.load_module(filename)
+ if backend == "vm":
+ rt_mod = session.get_function("runtime.Load_Executable")(code, rt_mod)
+ dev = session.device(dev_type=dev_type, dev_id=0)
+ nd_args = {k: ndarray.array(v, dev) for k, v in args.items()}
+ return continuation(rt_mod, dev, nd_args)
+ except Exception as exc: # pylint: disable=broad-except
+ print(
+ f"Run module {continuation.__name__} via RPC failed, exception: {exc}",
+ )
Review Comment:
Yeah it's too broad, I'll move that into continuation funciton.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 commented on pull request #11797: [MetaSchedule] Tuning Script Upgrade
Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on PR #11797:
URL: https://github.com/apache/tvm/pull/11797#issuecomment-1169056626
let's rebase against the lastest HEAD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 commented on a diff in pull request #11797: [MetaSchedule] Tuning Script Upgrade
Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on code in PR #11797:
URL: https://github.com/apache/tvm/pull/11797#discussion_r910457244
##########
python/tvm/meta_schedule/testing/utils.py:
##########
@@ -77,3 +81,145 @@ def apply_fixed_schedules(
database.commit_tuning_record(tune_rec)
return database
+
+
+def generate_input_data(input_shape: List[int], input_dtype: str) -> np.ndarray:
+ """Generate input date with given shape and data type.
+
+ Parameters
+ ----------
+ input_shape : List[int]
+ The shape of the input data.
+ input_dtype : str
+ The data type of the input date.
+
+ Returns
+ -------
+ input_data : np.ndarray
+ The generated input data with given shape and data type in numpy ndarray.
+ """
+ if input_dtype.startswith("float"):
+ return np.random.uniform(size=input_shape).astype(input_dtype)
+ if input_dtype in ["uint8", "int8"]:
+ return np.random.randint(
+ low=0,
+ high=127,
+ size=input_shape,
+ dtype="int32", # TODO(zxybazh): fix the datatype when int8 / uint8 is supported better
+ )
+ if input_dtype in ["int32", "int64"]:
+ return np.random.randint(low=0, high=10000, size=input_shape, dtype=input_dtype)
+ raise ValueError("Unsupported input datatype!")
+
+
+def f_timer(backend: str) -> Callable:
+ """Create a function to run and benchmark the performance of whole given runtime module,
+ or Executable in relay vm.
+
+ Parameters
+ ----------
+ backend : str
+ The backend to use, graph / vm.
+
+ Returns
+ -------
+ func : Callable
+ The function to benchmark the workload.
+ """
+
+ def f_timer_func(
+ rt_mod: Union[tvm.runtime.Module, tvm.runtime.vm.Executable],
+ dev: tvm.device,
+ input_data: Dict[str, NDArray],
+ ) -> None:
+ """Run and benchmark the given runtime module, print out the result.
+
+ Parameters
+ ----------
+ rt_mod : Union[tvm.runtime.Module, tvm.runtime.vm.Executable]
+ The runtime module or vm executable.
+ dev : tvm.device
+ The device type to run workload.
+ input_data : Dict[str, np.ndarray]
+ The input data as a dictionary.
+ """
+ from tvm.contrib.graph_executor import GraphModule # pylint:disable=import-outside-toplevel
+ from tvm.runtime.vm import VirtualMachine # pylint:disable=import-outside-toplevel
+
+ if backend == "vm":
+ vm = VirtualMachine(rt_mod, dev) # pylint: disable=invalid-name
+ ftimer = vm.benchmark(
+ dev, min_repeat_ms=500, repeat=5, number=1, end_to_end=False, **input_data
+ )
+ elif backend == "graph":
+ mod = GraphModule(rt_mod["default"](dev))
+ for input_name, input_value in input_data.items():
+ mod.set_input(input_name, input_value)
+ ftimer = mod.module.time_evaluator("run", dev, min_repeat_ms=500, repeat=5, number=1)()
+ else:
+ raise ValueError(f"Backend {backend} not supported in f_timer!")
+
+ results = list(np.array(ftimer.results) * 1000.0) # type: ignore
+
+ print("Running time in time_evaluator: ", results)
+ print("-------------------------------")
+ print(f" Min (ms) : {min(results)}")
+ print(f" Max (ms) : {max(results)}")
+ print(f" Median (ms) : {median(results)}")
+ print(f"Average (ms) : {sum(results) / len(results)}")
+
+ return f_timer_func
+
+
+def f_per_layer(graph: str) -> Callable:
Review Comment:
ditto. how about `create_per_layer_timer_func`
##########
python/tvm/meta_schedule/testing/utils.py:
##########
@@ -77,3 +81,145 @@ def apply_fixed_schedules(
database.commit_tuning_record(tune_rec)
return database
+
+
+def generate_input_data(input_shape: List[int], input_dtype: str) -> np.ndarray:
+ """Generate input date with given shape and data type.
+
+ Parameters
+ ----------
+ input_shape : List[int]
+ The shape of the input data.
+ input_dtype : str
+ The data type of the input date.
+
+ Returns
+ -------
+ input_data : np.ndarray
+ The generated input data with given shape and data type in numpy ndarray.
+ """
+ if input_dtype.startswith("float"):
+ return np.random.uniform(size=input_shape).astype(input_dtype)
+ if input_dtype in ["uint8", "int8"]:
+ return np.random.randint(
+ low=0,
+ high=127,
+ size=input_shape,
+ dtype="int32", # TODO(zxybazh): fix the datatype when int8 / uint8 is supported better
+ )
+ if input_dtype in ["int32", "int64"]:
+ return np.random.randint(low=0, high=10000, size=input_shape, dtype=input_dtype)
+ raise ValueError("Unsupported input datatype!")
+
+
+def f_timer(backend: str) -> Callable:
Review Comment:
let's find a better name. how about, `create_timer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 commented on a diff in pull request #11797: [MetaSchedule] Tuning Script Upgrade
Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on code in PR #11797:
URL: https://github.com/apache/tvm/pull/11797#discussion_r910453866
##########
python/tvm/meta_schedule/testing/custom_builder_runner.py:
##########
@@ -158,15 +160,22 @@ def run_module_via_rpc(
# pylint: enable=import-outside-toplevel
- with tempfile.TemporaryDirectory() as tmp_dir:
- filename = os.path.join(tmp_dir, "tvm_tmp_mod." + tar.output_format)
- lib.export_library(filename, tar)
- session = rpc_config.connect_server()
- session.upload(filename)
- _, filename = os.path.split(filename)
- rt_mod = session.load_module(filename)
- dev = session.device(dev_type=dev_type, dev_id=0)
- nd_args = {}
- for arg_key, arg_value in args.items():
- nd_args[arg_key] = ndarray.array(arg_value, dev)
- return continuation(rt_mod, dev, nd_args)
+ try:
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ filename = os.path.join(tmp_dir, "tvm_tmp_mod." + tar.output_format)
+ if backend == "vm":
+ code, lib = lib.save()
+ lib.export_library(filename, tar)
+ session = rpc_config.connect_server()
+ session.upload(filename)
+ _, filename = os.path.split(filename)
+ rt_mod = session.load_module(filename)
+ if backend == "vm":
+ rt_mod = session.get_function("runtime.Load_Executable")(code, rt_mod)
+ dev = session.device(dev_type=dev_type, dev_id=0)
+ nd_args = {k: ndarray.array(v, dev) for k, v in args.items()}
+ return continuation(rt_mod, dev, nd_args)
+ except Exception as exc: # pylint: disable=broad-except
+ print(
+ f"Run module {continuation.__name__} via RPC failed, exception: {exc}",
+ )
Review Comment:
quick question: why do we need to silence the exceptions here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zxybazh merged pull request #11797: [MetaSchedule] Tuning Script Upgrade
Posted by GitBox <gi...@apache.org>.
zxybazh merged PR #11797:
URL: https://github.com/apache/tvm/pull/11797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org