[GitHub] [tvm] lhutton1 commented on a change in pull request #9841: [microNPU] Add support for nearest neighbor and bilinear upsampling

[GitBox <gi...@apache.org>]

lhutton1 commented on a change in pull request #9841:
URL: https://github.com/apache/tvm/pull/9841#discussion_r784676873



##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -1145,6 +1145,94 @@ def split_pattern():
     return split
 
 
+class Resize2dParams:
+    """
+    This class will parse a call to ethos-u.resize2d composite function
+    and extract the parameter information.
+    """
+
+    composite_name = "ethos-u.resize2d"
+
+    def __init__(self, func_body: Call):
+        layout = "NHWC"
+
+        resize_2d = func_body
+        in_var = func_body.args[0]
+        if (
+            isinstance(resize_2d, tvm.relay.expr.Call)
+            and isinstance(resize_2d.op, tvm.ir.Op)
+            and resize_2d.op.name == "qnn.quantize"
+        ):
+            resize_2d = resize_2d.args[0]
+            in_var = in_var.args[0].args[0]
+        out_var = func_body
+
+        self.ifm = TensorParams(in_var, layout=layout)
+        self.ofm = TensorParams(out_var, layout=layout)
+
+        attrs = resize_2d.attrs
+        self.size = attrs.size
+        self.method = attrs.method
+        self.roi = attrs.roi

Review comment:
       "Region of Interest" that can be used to crop the input image. Apparently this can only be used when the `coordinate_transformation_mode` is `tf_crop_and_resize`.. potentially something to consider in the future

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -191,6 +191,29 @@ def get_range_for_dtype_str(dtype: str) -> Tuple[int, int]:
     return type_info.min, type_info.max
 
 
+def upscale_ofm(ofm_shape: List[int], upscale_height: int, upscale_width: int) -> List[int]:
+    """Calculate upscaled shape of the 4-dimensional Output Feature Map.
+
+    Parameters
+    ----------
+    ofm_shape : List[int]
+        The shape of the Output Feature Map.
+    upscale_height : int
+        The height of the upscaled output.
+    upscale_width : int
+        The width of the upscaled output.
+
+    Returns
+    -------
+    List[int]
+        Upscaled Output Feature Map shape.
+    """
+    assert len(ofm_shape) == 4, "Upscaling non 4-dimensional output is not supported."

Review comment:
       In TE where this function was used the tensors are always 4D and not brick format. This is removed in the latest revision.

##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -1145,6 +1145,94 @@ def split_pattern():
     return split
 
 
+class Resize2dParams:
+    """
+    This class will parse a call to ethos-u.resize2d composite function
+    and extract the parameter information.
+    """
+
+    composite_name = "ethos-u.resize2d"
+
+    def __init__(self, func_body: Call):
+        layout = "NHWC"
+
+        resize_2d = func_body
+        in_var = func_body.args[0]
+        if (
+            isinstance(resize_2d, tvm.relay.expr.Call)
+            and isinstance(resize_2d.op, tvm.ir.Op)
+            and resize_2d.op.name == "qnn.quantize"
+        ):
+            resize_2d = resize_2d.args[0]
+            in_var = in_var.args[0].args[0]
+        out_var = func_body
+
+        self.ifm = TensorParams(in_var, layout=layout)
+        self.ofm = TensorParams(out_var, layout=layout)
+
+        attrs = resize_2d.attrs
+        self.size = attrs.size
+        self.method = attrs.method
+        self.roi = attrs.roi
+        self.coordinate_transformation_mode = attrs.coordinate_transformation_mode
+        self.rounding_method = attrs.rounding_method
+        self.out_dtype = attrs.out_dtype
+
+    def is_valid(self) -> bool:
+        """
+        Checks whether image.resize2d has compatible attributes with HW.
+        """
+
+        def check_compatible_size(mode, method, upscale_size, height, width):
+            """Checking the provided upscale_size is compatible with the NPU. The NPU only
+            supports upsampling when the upsampling size is 2 * input_size, or when there is
+            no upsampling to be done, so check that this is the case. In the special case of
+            resize_bilinear with align_corners=True, the NPU only supports an upsampling
+            size of 2 * input_size - 1."""

Review comment:
       Yep, infact the compiler handles this case without any modification in the latest attempt, so we can remove the need for `upscale_height` and `upscale_width` altogether.

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -1242,6 +1242,114 @@ def __call__(self, *args, **kwargs):
         pass
 
 
+class Resize2dRewriter(DFPatternCallback):
+    """
+    Convert ethos-u.resize2d composite function to an equivalent operation that
+    performs the relevant upsampling operation.
+
+    Case 1: Upsample factor of 1 i.e. no upsampling:
+        add scalar of 0 - do nothing.

Review comment:
       Thanks that's correct, it was that before I changed my mind :) In the latest revision I removed support for "no up-sampling" entirely as I don't think it makes much sense, WDYT?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org