You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/01/05 10:25:54 UTC
[GitHub] [tvm] lhutton1 opened a new pull request #9841: [microNPU] Add support for nearest neighbor and bilinear upsampling
lhutton1 opened a new pull request #9841:
URL: https://github.com/apache/tvm/pull/9841
Adds support for 2x2 nearest neighbor and bilinear upsampling. In the case of bilinear upsampling with align_corners set to true, the upsampling size must be `2*input_size - 1` (as opposed to `2*input_size`).
cc @ekalda @mbaret @jacobbohlin @NicolaLancellotti @dchauhan-arm @manupa-arm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] manupa-arm commented on pull request #9841: [microNPU] Add support for nearest neighbor and bilinear upsampling
Posted by GitBox <gi...@apache.org>.
manupa-arm commented on pull request #9841:
URL: https://github.com/apache/tvm/pull/9841#issuecomment-1025941436
Thanks all! this is merged!.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] ekalda commented on a change in pull request #9841: [microNPU] Add support for nearest neighbor and bilinear upsampling
Posted by GitBox <gi...@apache.org>.
ekalda commented on a change in pull request #9841:
URL: https://github.com/apache/tvm/pull/9841#discussion_r784988584
##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -1145,6 +1145,94 @@ def split_pattern():
return split
+class Resize2dParams:
+ """
+ This class will parse a call to ethos-u.resize2d composite function
+ and extract the parameter information.
+ """
+
+ composite_name = "ethos-u.resize2d"
+
+ def __init__(self, func_body: Call):
+ layout = "NHWC"
+
+ resize_2d = func_body
+ in_var = func_body.args[0]
+ if (
+ isinstance(resize_2d, tvm.relay.expr.Call)
+ and isinstance(resize_2d.op, tvm.ir.Op)
+ and resize_2d.op.name == "qnn.quantize"
+ ):
+ resize_2d = resize_2d.args[0]
+ in_var = in_var.args[0].args[0]
+ out_var = func_body
+
+ self.ifm = TensorParams(in_var, layout=layout)
+ self.ofm = TensorParams(out_var, layout=layout)
+
+ attrs = resize_2d.attrs
+ self.size = attrs.size
+ self.method = attrs.method
+ self.roi = attrs.roi
Review comment:
Yeah that sounds non-trivial to support :D
##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -1145,6 +1145,94 @@ def split_pattern():
return split
+class Resize2dParams:
+ """
+ This class will parse a call to ethos-u.resize2d composite function
+ and extract the parameter information.
+ """
+
+ composite_name = "ethos-u.resize2d"
+
+ def __init__(self, func_body: Call):
+ layout = "NHWC"
+
+ resize_2d = func_body
+ in_var = func_body.args[0]
+ if (
+ isinstance(resize_2d, tvm.relay.expr.Call)
+ and isinstance(resize_2d.op, tvm.ir.Op)
+ and resize_2d.op.name == "qnn.quantize"
+ ):
+ resize_2d = resize_2d.args[0]
+ in_var = in_var.args[0].args[0]
+ out_var = func_body
+
+ self.ifm = TensorParams(in_var, layout=layout)
+ self.ofm = TensorParams(out_var, layout=layout)
+
+ attrs = resize_2d.attrs
+ self.size = attrs.size
+ self.method = attrs.method
+ self.roi = attrs.roi
+ self.coordinate_transformation_mode = attrs.coordinate_transformation_mode
+ self.rounding_method = attrs.rounding_method
+ self.out_dtype = attrs.out_dtype
+
+ def is_valid(self) -> bool:
+ """
+ Checks whether image.resize2d has compatible attributes with HW.
+ """
+
+ def check_compatible_size(mode, method, upscale_size, height, width):
+ """Checking the provided upscale_size is compatible with the NPU. The NPU only
+ supports upsampling when the upsampling size is 2 * input_size, or when there is
+ no upsampling to be done, so check that this is the case. In the special case of
+ resize_bilinear with align_corners=True, the NPU only supports an upsampling
+ size of 2 * input_size - 1."""
Review comment:
Woop cool!
##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -191,6 +191,29 @@ def get_range_for_dtype_str(dtype: str) -> Tuple[int, int]:
return type_info.min, type_info.max
+def upscale_ofm(ofm_shape: List[int], upscale_height: int, upscale_width: int) -> List[int]:
+ """Calculate upscaled shape of the 4-dimensional Output Feature Map.
+
+ Parameters
+ ----------
+ ofm_shape : List[int]
+ The shape of the Output Feature Map.
+ upscale_height : int
+ The height of the upscaled output.
+ upscale_width : int
+ The width of the upscaled output.
+
+ Returns
+ -------
+ List[int]
+ Upscaled Output Feature Map shape.
+ """
+ assert len(ofm_shape) == 4, "Upscaling non 4-dimensional output is not supported."
Review comment:
Ok cool! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] lhutton1 commented on a change in pull request #9841: [microNPU] Add support for nearest neighbor and bilinear upsampling
Posted by GitBox <gi...@apache.org>.
lhutton1 commented on a change in pull request #9841:
URL: https://github.com/apache/tvm/pull/9841#discussion_r784676873
##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -1145,6 +1145,94 @@ def split_pattern():
return split
+class Resize2dParams:
+ """
+ This class will parse a call to ethos-u.resize2d composite function
+ and extract the parameter information.
+ """
+
+ composite_name = "ethos-u.resize2d"
+
+ def __init__(self, func_body: Call):
+ layout = "NHWC"
+
+ resize_2d = func_body
+ in_var = func_body.args[0]
+ if (
+ isinstance(resize_2d, tvm.relay.expr.Call)
+ and isinstance(resize_2d.op, tvm.ir.Op)
+ and resize_2d.op.name == "qnn.quantize"
+ ):
+ resize_2d = resize_2d.args[0]
+ in_var = in_var.args[0].args[0]
+ out_var = func_body
+
+ self.ifm = TensorParams(in_var, layout=layout)
+ self.ofm = TensorParams(out_var, layout=layout)
+
+ attrs = resize_2d.attrs
+ self.size = attrs.size
+ self.method = attrs.method
+ self.roi = attrs.roi
Review comment:
"Region of Interest" that can be used to crop the input image. Apparently this can only be used when the `coordinate_transformation_mode` is `tf_crop_and_resize`.. potentially something to consider in the future
##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -191,6 +191,29 @@ def get_range_for_dtype_str(dtype: str) -> Tuple[int, int]:
return type_info.min, type_info.max
+def upscale_ofm(ofm_shape: List[int], upscale_height: int, upscale_width: int) -> List[int]:
+ """Calculate upscaled shape of the 4-dimensional Output Feature Map.
+
+ Parameters
+ ----------
+ ofm_shape : List[int]
+ The shape of the Output Feature Map.
+ upscale_height : int
+ The height of the upscaled output.
+ upscale_width : int
+ The width of the upscaled output.
+
+ Returns
+ -------
+ List[int]
+ Upscaled Output Feature Map shape.
+ """
+ assert len(ofm_shape) == 4, "Upscaling non 4-dimensional output is not supported."
Review comment:
In TE where this function was used the tensors are always 4D and not brick format. This is removed in the latest revision.
##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -1145,6 +1145,94 @@ def split_pattern():
return split
+class Resize2dParams:
+ """
+ This class will parse a call to ethos-u.resize2d composite function
+ and extract the parameter information.
+ """
+
+ composite_name = "ethos-u.resize2d"
+
+ def __init__(self, func_body: Call):
+ layout = "NHWC"
+
+ resize_2d = func_body
+ in_var = func_body.args[0]
+ if (
+ isinstance(resize_2d, tvm.relay.expr.Call)
+ and isinstance(resize_2d.op, tvm.ir.Op)
+ and resize_2d.op.name == "qnn.quantize"
+ ):
+ resize_2d = resize_2d.args[0]
+ in_var = in_var.args[0].args[0]
+ out_var = func_body
+
+ self.ifm = TensorParams(in_var, layout=layout)
+ self.ofm = TensorParams(out_var, layout=layout)
+
+ attrs = resize_2d.attrs
+ self.size = attrs.size
+ self.method = attrs.method
+ self.roi = attrs.roi
+ self.coordinate_transformation_mode = attrs.coordinate_transformation_mode
+ self.rounding_method = attrs.rounding_method
+ self.out_dtype = attrs.out_dtype
+
+ def is_valid(self) -> bool:
+ """
+ Checks whether image.resize2d has compatible attributes with HW.
+ """
+
+ def check_compatible_size(mode, method, upscale_size, height, width):
+ """Checking the provided upscale_size is compatible with the NPU. The NPU only
+ supports upsampling when the upsampling size is 2 * input_size, or when there is
+ no upsampling to be done, so check that this is the case. In the special case of
+ resize_bilinear with align_corners=True, the NPU only supports an upsampling
+ size of 2 * input_size - 1."""
Review comment:
Yep, infact the compiler handles this case without any modification in the latest attempt, so we can remove the need for `upscale_height` and `upscale_width` altogether.
##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -1242,6 +1242,114 @@ def __call__(self, *args, **kwargs):
pass
+class Resize2dRewriter(DFPatternCallback):
+ """
+ Convert ethos-u.resize2d composite function to an equivalent operation that
+ performs the relevant upsampling operation.
+
+ Case 1: Upsample factor of 1 i.e. no upsampling:
+ add scalar of 0 - do nothing.
Review comment:
Thanks that's correct, it was that before I changed my mind :) In the latest revision I removed support for "no up-sampling" entirely as I don't think it makes much sense, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] manupa-arm merged pull request #9841: [microNPU] Add support for nearest neighbor and bilinear upsampling
Posted by GitBox <gi...@apache.org>.
manupa-arm merged pull request #9841:
URL: https://github.com/apache/tvm/pull/9841
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] lhutton1 commented on pull request #9841: [microNPU] Add support for nearest neighbor and bilinear upsampling
Posted by GitBox <gi...@apache.org>.
lhutton1 commented on pull request #9841:
URL: https://github.com/apache/tvm/pull/9841#issuecomment-1014722575
As mentioned previously, a revised implementation has been pushed which removes the need for `upscale_height` and `upscale_width` to be passed to operator definitions, as well as correctly modeling the data dependency of upscaling in TE (previously not done), PTAL!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] ekalda commented on a change in pull request #9841: [microNPU] Add support for nearest neighbor and bilinear upsampling
Posted by GitBox <gi...@apache.org>.
ekalda commented on a change in pull request #9841:
URL: https://github.com/apache/tvm/pull/9841#discussion_r784149139
##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -191,6 +191,29 @@ def get_range_for_dtype_str(dtype: str) -> Tuple[int, int]:
return type_info.min, type_info.max
+def upscale_ofm(ofm_shape: List[int], upscale_height: int, upscale_width: int) -> List[int]:
+ """Calculate upscaled shape of the 4-dimensional Output Feature Map.
+
+ Parameters
+ ----------
+ ofm_shape : List[int]
+ The shape of the Output Feature Map.
+ upscale_height : int
+ The height of the upscaled output.
+ upscale_width : int
+ The width of the upscaled output.
+
+ Returns
+ -------
+ List[int]
+ Upscaled Output Feature Map shape.
+ """
+ assert len(ofm_shape) == 4, "Upscaling non 4-dimensional output is not supported."
Review comment:
What prevents us from not upscaling non 4D tensor?
##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -1145,6 +1145,94 @@ def split_pattern():
return split
+class Resize2dParams:
+ """
+ This class will parse a call to ethos-u.resize2d composite function
+ and extract the parameter information.
+ """
+
+ composite_name = "ethos-u.resize2d"
+
+ def __init__(self, func_body: Call):
+ layout = "NHWC"
+
+ resize_2d = func_body
+ in_var = func_body.args[0]
+ if (
+ isinstance(resize_2d, tvm.relay.expr.Call)
+ and isinstance(resize_2d.op, tvm.ir.Op)
+ and resize_2d.op.name == "qnn.quantize"
+ ):
+ resize_2d = resize_2d.args[0]
+ in_var = in_var.args[0].args[0]
+ out_var = func_body
+
+ self.ifm = TensorParams(in_var, layout=layout)
+ self.ofm = TensorParams(out_var, layout=layout)
+
+ attrs = resize_2d.attrs
+ self.size = attrs.size
+ self.method = attrs.method
+ self.roi = attrs.roi
Review comment:
For an enlightenment, what is roi?
##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -1242,6 +1242,114 @@ def __call__(self, *args, **kwargs):
pass
+class Resize2dRewriter(DFPatternCallback):
+ """
+ Convert ethos-u.resize2d composite function to an equivalent operation that
+ performs the relevant upsampling operation.
+
+ Case 1: Upsample factor of 1 i.e. no upsampling:
+ add scalar of 0 - do nothing.
Review comment:
Looks like you are doing an identity pooling instead of adding 0?
##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -1145,6 +1145,94 @@ def split_pattern():
return split
+class Resize2dParams:
+ """
+ This class will parse a call to ethos-u.resize2d composite function
+ and extract the parameter information.
+ """
+
+ composite_name = "ethos-u.resize2d"
+
+ def __init__(self, func_body: Call):
+ layout = "NHWC"
+
+ resize_2d = func_body
+ in_var = func_body.args[0]
+ if (
+ isinstance(resize_2d, tvm.relay.expr.Call)
+ and isinstance(resize_2d.op, tvm.ir.Op)
+ and resize_2d.op.name == "qnn.quantize"
+ ):
+ resize_2d = resize_2d.args[0]
+ in_var = in_var.args[0].args[0]
+ out_var = func_body
+
+ self.ifm = TensorParams(in_var, layout=layout)
+ self.ofm = TensorParams(out_var, layout=layout)
+
+ attrs = resize_2d.attrs
+ self.size = attrs.size
+ self.method = attrs.method
+ self.roi = attrs.roi
+ self.coordinate_transformation_mode = attrs.coordinate_transformation_mode
+ self.rounding_method = attrs.rounding_method
+ self.out_dtype = attrs.out_dtype
+
+ def is_valid(self) -> bool:
+ """
+ Checks whether image.resize2d has compatible attributes with HW.
+ """
+
+ def check_compatible_size(mode, method, upscale_size, height, width):
+ """Checking the provided upscale_size is compatible with the NPU. The NPU only
+ supports upsampling when the upsampling size is 2 * input_size, or when there is
+ no upsampling to be done, so check that this is the case. In the special case of
+ resize_bilinear with align_corners=True, the NPU only supports an upsampling
+ size of 2 * input_size - 1."""
Review comment:
Considering that we have these limitations, can't we always figure out the ofm size from the upsampling type and padding, without needing the `upscale_height` and `upscale_width` parameters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] lhutton1 commented on a change in pull request #9841: [microNPU] Add support for nearest neighbor and bilinear upsampling
Posted by GitBox <gi...@apache.org>.
lhutton1 commented on a change in pull request #9841:
URL: https://github.com/apache/tvm/pull/9841#discussion_r792512412
##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -1242,6 +1242,114 @@ def __call__(self, *args, **kwargs):
pass
+class Resize2dRewriter(DFPatternCallback):
+ """
+ Convert ethos-u.resize2d composite function to an equivalent operation that
+ performs the relevant upsampling operation.
+
+ Case 1: Upsample factor of 1 i.e. no upsampling:
+ add scalar of 0 - do nothing.
Review comment:
Updated, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org