You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/10/06 22:42:02 UTC
[GitHub] [tvm] tkonolige opened a new pull request, #13003: [ROOFLINE] Add support for different dtypes
tkonolige opened a new pull request, #13003:
URL: https://github.com/apache/tvm/pull/13003
Support different dtypes in roofline analysis. Only x86 support for now, but the interface is there to add support on cuda.
@AndrewZhaoLuo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] AndrewZhaoLuo merged pull request #13003: [ROOFLINE] Add support for different dtypes
Posted by GitBox <gi...@apache.org>.
AndrewZhaoLuo merged PR #13003:
URL: https://github.com/apache/tvm/pull/13003
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tkonolige commented on a diff in pull request #13003: [ROOFLINE] Add support for different dtypes
Posted by GitBox <gi...@apache.org>.
tkonolige commented on code in PR #13003:
URL: https://github.com/apache/tvm/pull/13003#discussion_r995201406
##########
python/tvm/utils/roofline/registry.py:
##########
@@ -15,18 +15,24 @@
# specific language governing permissions and limitations
# under the License.
"""Definition of generic functions for estimating peak flops and bandwidth"""
-from typing import Optional
-from ...target import Target, generic_func
-from ...runtime import Device
+from typing import Dict, Optional, Tuple
+
+import numpy as np
+
from ...rpc.client import RPCSession
+from ...runtime import Device
+from ...target import Target, generic_func
+from ...tir import PrimFunc
@generic_func
def estimate_peak_bandwidth(
+ func: PrimFunc,
Review Comment:
You might want to dispatch to different peak measurements based on the features or the primfunc. For example, I could detect if tensorcores could be used with the given primfunc or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] AndrewZhaoLuo commented on a diff in pull request #13003: [ROOFLINE] Add support for different dtypes
Posted by GitBox <gi...@apache.org>.
AndrewZhaoLuo commented on code in PR #13003:
URL: https://github.com/apache/tvm/pull/13003#discussion_r995220867
##########
python/tvm/utils/roofline/cuda.py:
##########
@@ -234,3 +259,63 @@ def estimate_peak_bandwidth(
b = nd.empty((blocks, 4, warp_size), dtype="float32", device=dev)
times = f.time_evaluator(f.entry_name, dev, repeat=10, number=1)(a, b)
return a.numpy().size * 4 / times.min # 4 bytes per float32
+
+
+@registry.estimate_peak_bandwidth.register("cuda")
+def estimate_peak_bandwidth(
+ func: PrimFunc, # pylint: disable=unused-argument
+ features: Dict[str, np.ndarray],
+ target: Target,
+ dev: Device,
+ remote: Optional[RPCSession] = None,
+) -> Tuple[float, float, str]:
+ """Estimate peak memory bandwidth of a target/device combo.
+
+ Peak bandwidth is estimated by running a small experiment on the underlying
+ hardware. The peak bandwidth measurement assumes that vector instructions
+ are being used to load the data.
+
+ Parameters
+ ----------
+ func : PrimFunc
+ Function to estimate peak bandwidth for. Used to check if a specific
+ kind of memory could be used with this function.
+ features : Dict[str, np.ndarry]
+ Features extracted from `func`. Used to check if a specific kind of
+ memory could be used with this function.
+ target : Target
+ Target to use for measurement. This target should be as specific to the
+ underlying hardware as possible.
+ dev : Device
+ Device to measure peak bandwidth on.
+ remote : Optional[RPCSession]
+ Remote session used to upload artifacts for runtime evaluation. Must be
+ the same session used to create `dev`.
+
+ Returns
+ -------
+ loaded_bytes : float
+ Estimated bytes loaded by `func`.
+ peak_bandwidth : float
+ Peak memory bandwidth in bytes/seconds.
+ name : str
+ Name of the memory being used.
+ """
+ loaded_bytes = 0.0
+ # assume no more than 100 buffers
+ for i in range(100):
Review Comment:
I would prefer the way I listed since it makes the least amount of assumptions.
Now it is highly unlikely we will come close to 100 buffers or ever have a case where we have "b1" and "b3" but not "b2", in general it is more robust and the more robust code is not more difficult to maintain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tkonolige commented on a diff in pull request #13003: [ROOFLINE] Add support for different dtypes
Posted by GitBox <gi...@apache.org>.
tkonolige commented on code in PR #13003:
URL: https://github.com/apache/tvm/pull/13003#discussion_r995202196
##########
python/tvm/utils/roofline/cuda.py:
##########
@@ -234,3 +259,63 @@ def estimate_peak_bandwidth(
b = nd.empty((blocks, 4, warp_size), dtype="float32", device=dev)
times = f.time_evaluator(f.entry_name, dev, repeat=10, number=1)(a, b)
return a.numpy().size * 4 / times.min # 4 bytes per float32
+
+
+@registry.estimate_peak_bandwidth.register("cuda")
+def estimate_peak_bandwidth(
Review Comment:
Renamed `estimate_peak_bandwidth_global` to `estimate_peak_bandwidth_global_mem`. Hopefully this makes it clear that `estimate_peak_bandwidth_global_mem` estimates the peak bandwidth to global memory. `estimate_peak_bandwidth` estimates the peak bandwidth depending on the primfunc and features.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tkonolige commented on a diff in pull request #13003: [ROOFLINE] Add support for different dtypes
Posted by GitBox <gi...@apache.org>.
tkonolige commented on code in PR #13003:
URL: https://github.com/apache/tvm/pull/13003#discussion_r995203574
##########
python/tvm/utils/roofline/cuda.py:
##########
@@ -234,3 +259,63 @@ def estimate_peak_bandwidth(
b = nd.empty((blocks, 4, warp_size), dtype="float32", device=dev)
times = f.time_evaluator(f.entry_name, dev, repeat=10, number=1)(a, b)
return a.numpy().size * 4 / times.min # 4 bytes per float32
+
+
+@registry.estimate_peak_bandwidth.register("cuda")
+def estimate_peak_bandwidth(
+ func: PrimFunc, # pylint: disable=unused-argument
+ features: Dict[str, np.ndarray],
+ target: Target,
+ dev: Device,
+ remote: Optional[RPCSession] = None,
+) -> Tuple[float, float, str]:
+ """Estimate peak memory bandwidth of a target/device combo.
+
+ Peak bandwidth is estimated by running a small experiment on the underlying
+ hardware. The peak bandwidth measurement assumes that vector instructions
+ are being used to load the data.
+
+ Parameters
+ ----------
+ func : PrimFunc
+ Function to estimate peak bandwidth for. Used to check if a specific
+ kind of memory could be used with this function.
+ features : Dict[str, np.ndarry]
+ Features extracted from `func`. Used to check if a specific kind of
+ memory could be used with this function.
+ target : Target
+ Target to use for measurement. This target should be as specific to the
+ underlying hardware as possible.
+ dev : Device
+ Device to measure peak bandwidth on.
+ remote : Optional[RPCSession]
+ Remote session used to upload artifacts for runtime evaluation. Must be
+ the same session used to create `dev`.
+
+ Returns
+ -------
+ loaded_bytes : float
+ Estimated bytes loaded by `func`.
+ peak_bandwidth : float
+ Peak memory bandwidth in bytes/seconds.
+ name : str
+ Name of the memory being used.
+ """
+ loaded_bytes = 0.0
+ # assume no more than 100 buffers
+ for i in range(100):
Review Comment:
That's an alternative way to do it. I've changed this to go until it doesn't find one. Does that work for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tkonolige commented on a diff in pull request #13003: [ROOFLINE] Add support for different dtypes
Posted by GitBox <gi...@apache.org>.
tkonolige commented on code in PR #13003:
URL: https://github.com/apache/tvm/pull/13003#discussion_r995200611
##########
python/tvm/utils/roofline/cuda.py:
##########
@@ -161,6 +165,51 @@ def peak_flops_tensorcore_tir(
return n * 16 * 16 * 16 * 2 * sms * 8 / times.min
+@registry.estimate_peak_flops.register("cuda")
+def estimate_peak_flops(
+ func: PrimFunc, # pylint: disable=unused-argument
+ features: Dict[str, np.ndarray],
+ target: Target,
+ dev: Device,
+ remote: Optional[RPCSession],
+) -> Tuple[float, float, str]:
+ """Estimate the peak FLOP/s of a cuda device.
+
+ Parameters
+ ----------
+ func : PrimFunc
+ Function to estimate peak flops for. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ features : Dict[str, np.ndarry]
+ Features extracted from `func`. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ target : Target
+ Target to run on. This should be as specific to the actual hardware as
+ possible.
+ dev : Device
+ Device to run on.
+ remote : Optional[RPCSession]
+ Remote session used to upload artifacts for runtime evaluation. Must be
+ the same session used to create `dev`.
+
+ Returns
+ -------
+ flops : float
+ Estimated number of flops used by `func`.
+ peak_flops : float
+ Approximate sustained FLOP/s of this target/device combo. Addition and
+ multiplications are each counted as separate FLOPs.
+ name : str
+ Dtype/intrinsic used by `func` to achieve peak flops.
+ """
+ assert nvcc.have_tensorcore(
+ dev.compute_version
+ ), "CUDA roofline only works with devices that have tensorcores"
+ flops = np.sum(features["float_addsub"] + features["float_mul"] + features["float_mad"])
+ peak_flops = estimate_peak_flops_tensorcore(target, dev, remote)
Review Comment:
Didn't have time for it in this PR, but the infrastructure is there to add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tkonolige commented on a diff in pull request #13003: [ROOFLINE] Add support for different dtypes
Posted by GitBox <gi...@apache.org>.
tkonolige commented on code in PR #13003:
URL: https://github.com/apache/tvm/pull/13003#discussion_r996099556
##########
python/tvm/utils/roofline/cuda.py:
##########
@@ -234,3 +259,63 @@ def estimate_peak_bandwidth(
b = nd.empty((blocks, 4, warp_size), dtype="float32", device=dev)
times = f.time_evaluator(f.entry_name, dev, repeat=10, number=1)(a, b)
return a.numpy().size * 4 / times.min # 4 bytes per float32
+
+
+@registry.estimate_peak_bandwidth.register("cuda")
+def estimate_peak_bandwidth(
+ func: PrimFunc, # pylint: disable=unused-argument
+ features: Dict[str, np.ndarray],
+ target: Target,
+ dev: Device,
+ remote: Optional[RPCSession] = None,
+) -> Tuple[float, float, str]:
+ """Estimate peak memory bandwidth of a target/device combo.
+
+ Peak bandwidth is estimated by running a small experiment on the underlying
+ hardware. The peak bandwidth measurement assumes that vector instructions
+ are being used to load the data.
+
+ Parameters
+ ----------
+ func : PrimFunc
+ Function to estimate peak bandwidth for. Used to check if a specific
+ kind of memory could be used with this function.
+ features : Dict[str, np.ndarry]
+ Features extracted from `func`. Used to check if a specific kind of
+ memory could be used with this function.
+ target : Target
+ Target to use for measurement. This target should be as specific to the
+ underlying hardware as possible.
+ dev : Device
+ Device to measure peak bandwidth on.
+ remote : Optional[RPCSession]
+ Remote session used to upload artifacts for runtime evaluation. Must be
+ the same session used to create `dev`.
+
+ Returns
+ -------
+ loaded_bytes : float
+ Estimated bytes loaded by `func`.
+ peak_bandwidth : float
+ Peak memory bandwidth in bytes/seconds.
+ name : str
+ Name of the memory being used.
+ """
+ loaded_bytes = 0.0
+ # assume no more than 100 buffers
+ for i in range(100):
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] AndrewZhaoLuo commented on a diff in pull request #13003: [ROOFLINE] Add support for different dtypes
Posted by GitBox <gi...@apache.org>.
AndrewZhaoLuo commented on code in PR #13003:
URL: https://github.com/apache/tvm/pull/13003#discussion_r995179709
##########
python/tvm/utils/roofline/cuda.py:
##########
@@ -161,6 +165,51 @@ def peak_flops_tensorcore_tir(
return n * 16 * 16 * 16 * 2 * sms * 8 / times.min
+@registry.estimate_peak_flops.register("cuda")
+def estimate_peak_flops(
+ func: PrimFunc, # pylint: disable=unused-argument
+ features: Dict[str, np.ndarray],
+ target: Target,
+ dev: Device,
+ remote: Optional[RPCSession],
+) -> Tuple[float, float, str]:
+ """Estimate the peak FLOP/s of a cuda device.
+
+ Parameters
+ ----------
+ func : PrimFunc
+ Function to estimate peak flops for. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ features : Dict[str, np.ndarry]
+ Features extracted from `func`. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ target : Target
+ Target to run on. This should be as specific to the actual hardware as
+ possible.
+ dev : Device
+ Device to run on.
+ remote : Optional[RPCSession]
+ Remote session used to upload artifacts for runtime evaluation. Must be
+ the same session used to create `dev`.
+
+ Returns
+ -------
+ flops : float
+ Estimated number of flops used by `func`.
+ peak_flops : float
+ Approximate sustained FLOP/s of this target/device combo. Addition and
+ multiplications are each counted as separate FLOPs.
+ name : str
+ Dtype/intrinsic used by `func` to achieve peak flops.
+ """
+ assert nvcc.have_tensorcore(
+ dev.compute_version
+ ), "CUDA roofline only works with devices that have tensorcores"
+ flops = np.sum(features["float_addsub"] + features["float_mul"] + features["float_mad"])
Review Comment:
do madd count as 2 FLOPS?
##########
python/tvm/utils/roofline/x86.py:
##########
@@ -155,12 +134,72 @@ def estimate_peak_fma_flops(
random_fill = get_global_func("tvm.contrib.random.random_fill")
assert random_fill, "Please make sure USE_RANDOM is ON in config.cmake"
- a = nd.empty((nthreads, num_vector_registers, vec_width), dtype="float32", device=dev)
+ a = nd.empty((nthreads, num_vector_registers, vec_width), dtype=dtype, device=dev)
random_fill(a)
times = f.time_evaluator(f.entry_name, dev, repeat=100, number=1)(a)
flops = 2 * vec_width * num_vector_registers * nthreads * iters # fma is two flops
- flop_s = flops / times.min
- return flop_s
+ return flops / times.min
+
+
+@registry.estimate_peak_flops.register("cpu")
+def estimate_peak_fma_flops(
+ func: PrimFunc,
+ features: Dict[str, np.ndarray],
+ target: Target,
+ dev: Device,
+ remote: Optional[RPCSession],
+ vec_width: Optional[int] = None,
+ num_vector_registers: Optional[int] = None,
+) -> Tuple[float, float, str]:
+ """
+ Estimate the maximum number of FLOP/s this target/device combo is capable
+ of reaching by running a test program. This assumes vectorized FMA
+ (fused-multiply-add) instructions.
+
+
+ Parameters
+ ----------
+ func : PrimFunc
+ Function to estimate peak flops for. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ features : Dict[str, np.ndarry]
+ Features extracted from `func`. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ target : Target
+ Target to run on. This should be as specific to the actual hardware as
+ possible to make sure that LLVM generates the best vector code.
+ dev : Device
+ Device to run on.
+ remote : Optional[RPCSession]
+ Remote session used to upload artifacts for runtime evaluation. Must be
+ the same session used to create `dev`.
+ vec_width : Optional[int]
+ Vector width of SIMD units on the underlying hardware. Will try to
+ infer if no value is provided.
+ num_vector_registers : Optional[int]
+ Number of vector registers on the underlying hardware. Will try to
+ infer if no value is provided.
+
+ Returns
+ -------
+ flops : float
+ Estimated number of flops used by `func`.
+ peak_flops : float
+ Approximate sustained FLOP/s of this target/device combo assuming
+ vectorized FMA instructions. Each FMA operation counts as two FLOPs.
+ name : str
+ Dtype/intrinsic used by `func` to achieve peak flops.
+ """
+ # assume that the first argument's dtype is the one we want
+ dtype = list(func.buffer_map.values())[0].dtype
+ if "int" in dtype:
+ flops = np.sum(features["int_addsub"] + features["int_mul"] + features["int_mad"])
Review Comment:
should we count mad as 2 flops here, we seem to count fma above as 2.
##########
python/tvm/utils/roofline/cuda.py:
##########
@@ -161,6 +165,51 @@ def peak_flops_tensorcore_tir(
return n * 16 * 16 * 16 * 2 * sms * 8 / times.min
+@registry.estimate_peak_flops.register("cuda")
+def estimate_peak_flops(
+ func: PrimFunc, # pylint: disable=unused-argument
+ features: Dict[str, np.ndarray],
+ target: Target,
+ dev: Device,
+ remote: Optional[RPCSession],
+) -> Tuple[float, float, str]:
+ """Estimate the peak FLOP/s of a cuda device.
+
+ Parameters
+ ----------
+ func : PrimFunc
+ Function to estimate peak flops for. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ features : Dict[str, np.ndarry]
+ Features extracted from `func`. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ target : Target
+ Target to run on. This should be as specific to the actual hardware as
+ possible.
+ dev : Device
+ Device to run on.
+ remote : Optional[RPCSession]
+ Remote session used to upload artifacts for runtime evaluation. Must be
+ the same session used to create `dev`.
+
+ Returns
+ -------
+ flops : float
+ Estimated number of flops used by `func`.
+ peak_flops : float
+ Approximate sustained FLOP/s of this target/device combo. Addition and
+ multiplications are each counted as separate FLOPs.
+ name : str
+ Dtype/intrinsic used by `func` to achieve peak flops.
+ """
+ assert nvcc.have_tensorcore(
+ dev.compute_version
+ ), "CUDA roofline only works with devices that have tensorcores"
+ flops = np.sum(features["float_addsub"] + features["float_mul"] + features["float_mad"])
Review Comment:
is division not usually included in FLOP calculations?
##########
python/tvm/utils/roofline/cuda.py:
##########
@@ -234,3 +259,63 @@ def estimate_peak_bandwidth(
b = nd.empty((blocks, 4, warp_size), dtype="float32", device=dev)
times = f.time_evaluator(f.entry_name, dev, repeat=10, number=1)(a, b)
return a.numpy().size * 4 / times.min # 4 bytes per float32
+
+
+@registry.estimate_peak_bandwidth.register("cuda")
+def estimate_peak_bandwidth(
Review Comment:
Naming with the new scheme is a little confusing, hard to delineate `estimate_peak_bandwidth_global` and `estimate_peak_bandwidth` and with the other new functions, can you make a better name?
##########
python/tvm/utils/roofline/cuda.py:
##########
@@ -161,6 +165,51 @@ def peak_flops_tensorcore_tir(
return n * 16 * 16 * 16 * 2 * sms * 8 / times.min
+@registry.estimate_peak_flops.register("cuda")
+def estimate_peak_flops(
+ func: PrimFunc, # pylint: disable=unused-argument
+ features: Dict[str, np.ndarray],
+ target: Target,
+ dev: Device,
+ remote: Optional[RPCSession],
+) -> Tuple[float, float, str]:
+ """Estimate the peak FLOP/s of a cuda device.
+
+ Parameters
+ ----------
+ func : PrimFunc
+ Function to estimate peak flops for. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ features : Dict[str, np.ndarry]
+ Features extracted from `func`. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ target : Target
+ Target to run on. This should be as specific to the actual hardware as
+ possible.
+ dev : Device
+ Device to run on.
+ remote : Optional[RPCSession]
+ Remote session used to upload artifacts for runtime evaluation. Must be
+ the same session used to create `dev`.
+
+ Returns
+ -------
+ flops : float
+ Estimated number of flops used by `func`.
+ peak_flops : float
+ Approximate sustained FLOP/s of this target/device combo. Addition and
+ multiplications are each counted as separate FLOPs.
+ name : str
+ Dtype/intrinsic used by `func` to achieve peak flops.
+ """
+ assert nvcc.have_tensorcore(
+ dev.compute_version
+ ), "CUDA roofline only works with devices that have tensorcores"
+ flops = np.sum(features["float_addsub"] + features["float_mul"] + features["float_mad"])
+ peak_flops = estimate_peak_flops_tensorcore(target, dev, remote)
Review Comment:
Is this ready to hook up INT8 peak flop calculations too? (might be useful for Quantized model on tensorcore)
##########
python/tvm/utils/roofline/cuda.py:
##########
@@ -234,3 +259,63 @@ def estimate_peak_bandwidth(
b = nd.empty((blocks, 4, warp_size), dtype="float32", device=dev)
times = f.time_evaluator(f.entry_name, dev, repeat=10, number=1)(a, b)
return a.numpy().size * 4 / times.min # 4 bytes per float32
+
+
+@registry.estimate_peak_bandwidth.register("cuda")
+def estimate_peak_bandwidth(
+ func: PrimFunc, # pylint: disable=unused-argument
+ features: Dict[str, np.ndarray],
+ target: Target,
+ dev: Device,
+ remote: Optional[RPCSession] = None,
+) -> Tuple[float, float, str]:
+ """Estimate peak memory bandwidth of a target/device combo.
+
+ Peak bandwidth is estimated by running a small experiment on the underlying
+ hardware. The peak bandwidth measurement assumes that vector instructions
+ are being used to load the data.
+
+ Parameters
+ ----------
+ func : PrimFunc
+ Function to estimate peak bandwidth for. Used to check if a specific
+ kind of memory could be used with this function.
+ features : Dict[str, np.ndarry]
+ Features extracted from `func`. Used to check if a specific kind of
+ memory could be used with this function.
+ target : Target
+ Target to use for measurement. This target should be as specific to the
+ underlying hardware as possible.
+ dev : Device
+ Device to measure peak bandwidth on.
+ remote : Optional[RPCSession]
+ Remote session used to upload artifacts for runtime evaluation. Must be
+ the same session used to create `dev`.
+
+ Returns
+ -------
+ loaded_bytes : float
+ Estimated bytes loaded by `func`.
+ peak_bandwidth : float
+ Peak memory bandwidth in bytes/seconds.
+ name : str
+ Name of the memory being used.
+ """
+ loaded_bytes = 0.0
+ # assume no more than 100 buffers
+ for i in range(100):
Review Comment:
Can we just iterate through all features and find the ones which match the pattern?
##########
python/tvm/utils/roofline/registry.py:
##########
@@ -15,18 +15,24 @@
# specific language governing permissions and limitations
# under the License.
"""Definition of generic functions for estimating peak flops and bandwidth"""
-from typing import Optional
-from ...target import Target, generic_func
-from ...runtime import Device
+from typing import Dict, Optional, Tuple
+
+import numpy as np
+
from ...rpc.client import RPCSession
+from ...runtime import Device
+from ...target import Target, generic_func
+from ...tir import PrimFunc
@generic_func
def estimate_peak_bandwidth(
+ func: PrimFunc,
Review Comment:
hmm what necessitates this change in interface. It seems to me `func` and `features` (the added inputs) are always used to calculate the added float and str in the return type.
Why not just do the old thing and split up the calls to get the peak statistic, and one to get the actual statistics?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tkonolige commented on a diff in pull request #13003: [ROOFLINE] Add support for different dtypes
Posted by GitBox <gi...@apache.org>.
tkonolige commented on code in PR #13003:
URL: https://github.com/apache/tvm/pull/13003#discussion_r995200316
##########
python/tvm/utils/roofline/cuda.py:
##########
@@ -161,6 +165,51 @@ def peak_flops_tensorcore_tir(
return n * 16 * 16 * 16 * 2 * sms * 8 / times.min
+@registry.estimate_peak_flops.register("cuda")
+def estimate_peak_flops(
+ func: PrimFunc, # pylint: disable=unused-argument
+ features: Dict[str, np.ndarray],
+ target: Target,
+ dev: Device,
+ remote: Optional[RPCSession],
+) -> Tuple[float, float, str]:
+ """Estimate the peak FLOP/s of a cuda device.
+
+ Parameters
+ ----------
+ func : PrimFunc
+ Function to estimate peak flops for. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ features : Dict[str, np.ndarry]
+ Features extracted from `func`. Used to check if a specific kind
+ intrinsic or dtype could be used with this function.
+ target : Target
+ Target to run on. This should be as specific to the actual hardware as
+ possible.
+ dev : Device
+ Device to run on.
+ remote : Optional[RPCSession]
+ Remote session used to upload artifacts for runtime evaluation. Must be
+ the same session used to create `dev`.
+
+ Returns
+ -------
+ flops : float
+ Estimated number of flops used by `func`.
+ peak_flops : float
+ Approximate sustained FLOP/s of this target/device combo. Addition and
+ multiplications are each counted as separate FLOPs.
+ name : str
+ Dtype/intrinsic used by `func` to achieve peak flops.
+ """
+ assert nvcc.have_tensorcore(
+ dev.compute_version
+ ), "CUDA roofline only works with devices that have tensorcores"
+ flops = np.sum(features["float_addsub"] + features["float_mul"] + features["float_mad"])
Review Comment:
Good points, added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tvm-bot commented on pull request #13003: [ROOFLINE] Add support for different dtypes
Posted by GitBox <gi...@apache.org>.
tvm-bot commented on PR #13003:
URL: https://github.com/apache/tvm/pull/13003#issuecomment-1275466948
<!---bot-comment-->
Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @-ing them in a comment.
<!--bot-comment-docs-start-->
* Built docs for commit cf3b3aaa441500b4bcd590c9cc88641deaab46db can be found [here](https://pr-docs.tlcpack.ai/PR-13003/2/docs/index.html).<!--bot-comment-docs-end-->
<sub>Generated by [tvm-bot](https://github.com/apache/tvm/blob/main/ci/README.md#github-actions)</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] AndrewZhaoLuo commented on pull request #13003: [ROOFLINE] Add support for different dtypes
Posted by GitBox <gi...@apache.org>.
AndrewZhaoLuo commented on PR #13003:
URL: https://github.com/apache/tvm/pull/13003#issuecomment-1273632991
Will take a look today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org