[GitHub] [tvm] ekalda commented on a change in pull request #9841: [microNPU] Add support for nearest neighbor and bilinear upsampling

[GitBox <gi...@apache.org>]

ekalda commented on a change in pull request #9841:
URL: https://github.com/apache/tvm/pull/9841#discussion_r784149139



##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -191,6 +191,29 @@ def get_range_for_dtype_str(dtype: str) -> Tuple[int, int]:
     return type_info.min, type_info.max
 
 
+def upscale_ofm(ofm_shape: List[int], upscale_height: int, upscale_width: int) -> List[int]:
+    """Calculate upscaled shape of the 4-dimensional Output Feature Map.
+
+    Parameters
+    ----------
+    ofm_shape : List[int]
+        The shape of the Output Feature Map.
+    upscale_height : int
+        The height of the upscaled output.
+    upscale_width : int
+        The width of the upscaled output.
+
+    Returns
+    -------
+    List[int]
+        Upscaled Output Feature Map shape.
+    """
+    assert len(ofm_shape) == 4, "Upscaling non 4-dimensional output is not supported."

Review comment:
       What prevents us from not upscaling non 4D tensor?

##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -1145,6 +1145,94 @@ def split_pattern():
     return split
 
 
+class Resize2dParams:
+    """
+    This class will parse a call to ethos-u.resize2d composite function
+    and extract the parameter information.
+    """
+
+    composite_name = "ethos-u.resize2d"
+
+    def __init__(self, func_body: Call):
+        layout = "NHWC"
+
+        resize_2d = func_body
+        in_var = func_body.args[0]
+        if (
+            isinstance(resize_2d, tvm.relay.expr.Call)
+            and isinstance(resize_2d.op, tvm.ir.Op)
+            and resize_2d.op.name == "qnn.quantize"
+        ):
+            resize_2d = resize_2d.args[0]
+            in_var = in_var.args[0].args[0]
+        out_var = func_body
+
+        self.ifm = TensorParams(in_var, layout=layout)
+        self.ofm = TensorParams(out_var, layout=layout)
+
+        attrs = resize_2d.attrs
+        self.size = attrs.size
+        self.method = attrs.method
+        self.roi = attrs.roi

Review comment:
       For an enlightenment, what is roi?

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -1242,6 +1242,114 @@ def __call__(self, *args, **kwargs):
         pass
 
 
+class Resize2dRewriter(DFPatternCallback):
+    """
+    Convert ethos-u.resize2d composite function to an equivalent operation that
+    performs the relevant upsampling operation.
+
+    Case 1: Upsample factor of 1 i.e. no upsampling:
+        add scalar of 0 - do nothing.

Review comment:
       Looks like you are doing an identity pooling instead of adding 0?

##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -1145,6 +1145,94 @@ def split_pattern():
     return split
 
 
+class Resize2dParams:
+    """
+    This class will parse a call to ethos-u.resize2d composite function
+    and extract the parameter information.
+    """
+
+    composite_name = "ethos-u.resize2d"
+
+    def __init__(self, func_body: Call):
+        layout = "NHWC"
+
+        resize_2d = func_body
+        in_var = func_body.args[0]
+        if (
+            isinstance(resize_2d, tvm.relay.expr.Call)
+            and isinstance(resize_2d.op, tvm.ir.Op)
+            and resize_2d.op.name == "qnn.quantize"
+        ):
+            resize_2d = resize_2d.args[0]
+            in_var = in_var.args[0].args[0]
+        out_var = func_body
+
+        self.ifm = TensorParams(in_var, layout=layout)
+        self.ofm = TensorParams(out_var, layout=layout)
+
+        attrs = resize_2d.attrs
+        self.size = attrs.size
+        self.method = attrs.method
+        self.roi = attrs.roi
+        self.coordinate_transformation_mode = attrs.coordinate_transformation_mode
+        self.rounding_method = attrs.rounding_method
+        self.out_dtype = attrs.out_dtype
+
+    def is_valid(self) -> bool:
+        """
+        Checks whether image.resize2d has compatible attributes with HW.
+        """
+
+        def check_compatible_size(mode, method, upscale_size, height, width):
+            """Checking the provided upscale_size is compatible with the NPU. The NPU only
+            supports upsampling when the upsampling size is 2 * input_size, or when there is
+            no upsampling to be done, so check that this is the case. In the special case of
+            resize_bilinear with align_corners=True, the NPU only supports an upsampling
+            size of 2 * input_size - 1."""

Review comment:
       Considering that we have these limitations, can't we always figure out the ofm size from the upsampling type and padding, without needing the `upscale_height` and `upscale_width` parameters?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org