You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/08/19 14:39:53 UTC

[GitHub] [tvm] manupa-arm opened a new pull request #8795: Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

manupa-arm opened a new pull request #8795:
URL: https://github.com/apache/tvm/pull/8795


   This commit adds mainly the relay passes and ethosu_conv2d
   operator to relay. The relay passes include the legalizations
   and preprocessing of the relay graph in the paritioning.
   
   
   Co-authored-by: Matthew Barrett <ma...@arm.com>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-903770725


   Also pinging some Arm people for review: @ekalda @NicolaLancellotti @grant-arm @Leo-arm @tristan-arm @u99127


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-911940455


   Thanks @Mousius  -- GH kept some comments hidden from me.
   
   @jroesch , sorry now only got to finish the round.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] jroesch commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
jroesch commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-904289451


   cc @mbs-octoml @mikepapadim @electriclilies @csullivan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r701628365



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,218 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel, no-value-for-parameter
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm import ir
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """This rewriting converts split operations into a sequence of
+    strided_slice operations, because codegen is going to be based
+    on strided_slices that will define the slice of the tensor that
+    will be fed to the consumer.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split operator takes an array of indices or an integer
+        indicating the number of splits. However, its an array of indices could
+        represent both cases, therefore this function just make it an array of
+        indices where each index represent the co-ordinate of beginning of each
+        section -- defines as section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The Relay Call expression for a split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+@ir.transform.module_pass(opt_level=1)
+class LegalizeSplit:
+    """This is the pass that wraps SplitRewriter"""
+
+    def transform_module(self, mod, ctx):
+        for gv, func in mod.functions.items():
+            func = rewrite(SplitRewriter(), func)
+            mod.update_func(gv, func)
+        return mod
+
+    def __call__(self, *args, **kwargs):

Review comment:
       We discussed. summary is linter gets confused as it is unable to figure out the decorator makes this callable, therefore keep this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r701628485



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,218 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel, no-value-for-parameter
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm import ir
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """This rewriting converts split operations into a sequence of
+    strided_slice operations, because codegen is going to be based
+    on strided_slices that will define the slice of the tensor that
+    will be fed to the consumer.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split operator takes an array of indices or an integer
+        indicating the number of splits. However, its an array of indices could
+        represent both cases, therefore this function just make it an array of
+        indices where each index represent the co-ordinate of beginning of each
+        section -- defines as section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The Relay Call expression for a split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+@ir.transform.module_pass(opt_level=1)
+class LegalizeSplit:
+    """This is the pass that wraps SplitRewriter"""
+
+    def transform_module(self, mod, ctx):
+        for gv, func in mod.functions.items():
+            func = rewrite(SplitRewriter(), func)
+            mod.update_func(gv, func)
+        return mod
+
+    def __call__(self, *args, **kwargs):
+        pass
+
+
+class EthosUConv2DRewriter(DFPatternCallback):
+    """Convert conv2d related composite functions into ethosu_conv2d operators"""
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.pattern = (wildcard().has_attr({"Composite": "ethosu.qnn_conv2d"}))(wildcard())
+
+    def callback(self, pre, post, node_map):
+        params = ethosu_patterns.QnnConv2DParams(post.op.body)
+        params.ifm.tensor = post.args[0]
+        channels_map = {
+            "NHWC": 3,
+        }
+        if str(params.ofm.layout) not in channels_map.keys():
+            raise UnsupportedLayout(str(params.ofm.layout))
+        kernel_size_map = {
+            "HWIO": params.weights.shape[0:2],
+            "OHWI": params.weights.shape[1:3],
+            "HWOI": params.weights.shape[0:2],
+        }
+        if str(params.weights.layout) not in kernel_size_map.keys():
+            raise UnsupportedLayout(str(params.weights.layout))
+        activation_map = {"clip": "CLIP"}
+        weight_to_ohwi_transform_map = {"HWIO": [3, 0, 1, 2]}
+        weights_values = params.weights.values
+        weights_values_ohwi = np.transpose(
+            weights_values, weight_to_ohwi_transform_map[str(params.weights.layout)]
+        )
+        if params.activation:
+            activation = activation_map[params.activation.op.name]
+            clip_min = int(params.activation.attrs.a_min)
+            clip_max = int(params.activation.attrs.a_max)
+        else:
+            activation = "NONE"
+            clip_min = 0
+            clip_max = 0
+        scale_bias = vela_api.pack_biases(
+            biases=params.biases.tensor.data.asnumpy(),
+            ifm_scale=params.ifm.q_params.scale_f32,
+            ifm_dtype=np.dtype(params.ifm.dtype),
+            weight_scales=params.weights.q_params.scale_f32,
+            ofm_scale=params.ofm.q_params.scale_f32,
+            is_activation_tanh_or_sigmoid=activation in ["TANH", "SIGMOID"],
+        )
+        ethosu_conv2d = ethosu_ops.ethosu_conv2d(
+            ifm=post.args[0],
+            weight=relay.const(weights_values_ohwi, params.weights.values.dtype),
+            scale_bias=relay.const(scale_bias, "uint8"),
+            lut=relay.const([], dtype="int8"),
+            ifm_scale=float(params.ifm.q_params.scale_f32),
+            ifm_zero_point=int(params.ifm.q_params.zero_point),
+            weight_zero_point=int(params.weights.q_params.zero_point),
+            ofm_scale=float(params.ofm.q_params.scale_f32),
+            ofm_zero_point=int(params.ofm.q_params.zero_point),
+            kernel_shape=kernel_size_map[str(params.weights.layout)],
+            ofm_channels=params.ofm.shape[channels_map[str(params.ofm.layout)]],
+            strides=params.strides,
+            padding=params.padding,
+            dilation=params.dilation,
+            activation=activation,
+            clip_min=clip_min,
+            clip_max=clip_max,
+            upscale="NONE",
+            ifm_layout=str(params.ifm.layout),
+            ofm_layout=str(params.ofm.layout),
+        )
+        return ethosu_conv2d
+
+
+@ir.transform.module_pass(opt_level=1)
+class LegalizeEthosUConv2D:
+    """This is the pass that wraps the EthosUConv2DRewriter"""
+
+    def transform_module(self, mod, ctx):
+        for gv, func in mod.functions.items():
+            func = rewrite(EthosUConv2DRewriter(), func)
+            mod.update_func(gv, func)
+        return mod
+
+    def __call__(self, *args, **kwargs):

Review comment:
       same anwser as above




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbs-octoml commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-914411006


   LGTM thanks for the type annots, does it work if you enable checking in the mypy.ini? If so let's include that in this pr.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] jroesch merged pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
jroesch merged pull request #8795:
URL: https://github.com/apache/tvm/pull/8795


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695434506



##########
File path: python/tvm/relay/backend/contrib/ethosu/errors.py
##########
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=super-init-not-called
+"""This module is to hold all type of errors associated Arm(R) Ethos(TM)-U NPU Codegen"""

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695571625



##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU supported operators."""
+import numpy as np
+
+from tvm.relay.expr import Constant
+from tvm.relay.op.contrib.register import register_pattern_table
+from tvm.relay.dataflow_pattern import wildcard, is_op, is_constant
+from tvm.relay.backend.contrib.ethosu.util import QConv2DArgs
+from tvm.relay.backend.contrib.ethosu.util import BiasAddArgs
+from tvm.relay.backend.contrib.ethosu.util import RequantArgs
+from tvm.relay.backend.contrib.ethosu.util import get_dim_value
+from ethosu.vela import api as vapi
+
+
+def check_strides(strides):
+    """Checks whether strides are within the limits supported by the hardware"""
+    stride_range = (1, 3)
+    smin, smax = stride_range
+    if not smax >= strides[0] >= smin:
+        return False
+    if not smax >= strides[1] >= smin:
+        return False
+    return True
+
+
+def check_valid_dtypes(tensor_params):
+    """Check whether dtypes are supported by the hardware"""
+    supported_dtypes = (np.uint8, np.int8)
+    for tep in tensor_params:
+        # Check for dtypes
+        if np.dtype(tep.dtype) not in supported_dtypes:
+            return False
+        # Check for shape sizes
+        if any(dimlen > 65536 for dimlen in tep.shape):
+            return False
+    return True
+
+
+def check_weights(weights, dilation):
+    """Checks whether weight tensor is compatible with HW"""
+    dilated_height_range = (1, 64)
+    dilated_hxw_range = (1, 64 * 64)
+    weights_limit = 127 * 65536
+    dilated_width = (weights.shape[get_dim_value(weights.layout, "W")] - 1) * dilation[0] + 1
+    dilated_height = (weights.shape[get_dim_value(weights.layout, "H")] - 1) * dilation[1] + 1
+    dh_min, dh_max = dilated_height_range
+    if not dh_min <= dilated_height <= dh_max:
+        return False
+    dilated_hxw = dilated_height * dilated_width
+    dhxw_min, dhxw_max = dilated_hxw_range
+    if not dhxw_min <= dilated_hxw <= dhxw_max:
+        return False
+    # A saturation upper bound check for accumulators
+    weights.values = weights.values - weights.q_params.zero_point
+    axis = (
+        get_dim_value(weights.layout, "H"),
+        get_dim_value(weights.layout, "W"),
+        get_dim_value(weights.layout, "I"),
+    )
+    sum_weights = np.amax(np.sum(np.absolute(weights.values), axis=axis))
+    if not sum_weights <= weights_limit:
+        return False
+    return True
+
+
+def check_bias(bias):
+    """Check whether the bias values fit in 40 bits"""
+    if bias and bias.dtype == np.dtype("int64"):
+        valid = all(len(bin(bias_value)[2:]) <= 40 for bias_value in bias.values)
+        return valid
+    return True
+
+
+def check_batch_size(ifm):
+    """Checks for the number of batches vela currently supports"""
+    if ifm.shape[0] != 1:
+        return False
+    return True
+
+
+def check_dilation(dilation):
+    """Checks whether dilation is within the limits supported by the hardware"""
+    dilation_range = (1, 2)
+    dmin, dmax = dilation_range
+    if not dmin <= dilation[0] <= dmax:
+        return False
+    if not dmin <= dilation[1] <= dmax:
+        return False
+    return True
+
+
+def check_padding(padding, bounds):
+    """Checks whether padding is within the limits supported by the hardware"""
+    if len(padding) != 4 or len(bounds) != 4:
+        return False
+    top, left, bottom, right = padding
+    topb, leftb, bottomb, rightb = bounds
+    if top > topb or left > leftb or bottom > bottomb or right > rightb:
+        return False
+    return True
+
+
+class TensorParams:
+    """
+    This class will parse a tvm Expr along with quantization scale
+    and zero point to populate parameters that are required
+    for the creation of tensors in Vela.
+    """
+
+    def __init__(self, tensor, layout=None, scale=None, zero_point=None):
+        self.tensor = tensor
+        if isinstance(tensor, Constant):
+            self.values = tensor.data.asnumpy()
+        else:
+            self.values = None
+        self.dtype = tensor.checked_type.dtype
+        self.shape = [int(i) for i in tensor.checked_type.shape]
+        self.layout = layout
+
+        if scale is not None and zero_point is not None:
+            self.q_params = vapi.NpuQuantization(
+                scale.data.asnumpy().astype("float32"), zero_point.data.asnumpy().astype(self.dtype)
+            )
+        else:
+            # put default values
+            self.q_params = vapi.NpuQuantization(1.0, 0)
+
+
+class QnnConv2DParams:
+    """
+    This class will parse a Call to a ethosu.qnn_conv2d_clip composite function

Review comment:
       Yes, and corrected




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] jroesch commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
jroesch commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r694460440



##########
File path: python/tvm/relay/backend/contrib/ethosu/__init__.py
##########
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU codegen modules for relay."""

Review comment:
       Nit: captilization

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,

Review comment:
       ```suggestion
       """This rewriting converts split operations into a sequence of strided_slice operations,
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/preprocess.py
##########
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+"""Set of passes to pre-process the IRModule prior to codegen"""
+from . import _ffi_api
+
+
+def preprocess_ext_io():

Review comment:
       API document string, what does this return? 

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The relay expression for split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+class EthosUConv2DRewriter(DFPatternCallback):
+    """Convert conv2d related composite functions to ethosu_conv2d operators"""
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.pattern = (wildcard().has_attr({"Composite": "ethosu.qnn_conv2d"}))(wildcard())
+
+    def callback(self, pre, post, node_map):
+        params = ethosu_patterns.QnnConv2DParams(post.op.body)
+        params.ifm.tensor = post.args[0]
+        channels_map = {
+            "NHWC": 3,
+        }
+        if str(params.ofm.layout) not in channels_map.keys():
+            raise UnsupportedLayout(str(params.ofm.layout))
+        kernel_size_map = {
+            "HWIO": params.weights.shape[0:2],
+            "OHWI": params.weights.shape[1:3],
+            "HWOI": params.weights.shape[0:2],
+        }
+        if str(params.weights.layout) not in kernel_size_map.keys():
+            raise UnsupportedLayout(str(params.weights.layout))
+        activation_map = {"clip": "CLIP"}
+        weight_to_ohwi_transform_map = {"HWIO": [3, 0, 1, 2]}
+        weights_values = params.weights.values
+        weights_values_ohwi = np.transpose(
+            weights_values, weight_to_ohwi_transform_map[str(params.weights.layout)]
+        )
+        if params.activation:
+            activation = activation_map[params.activation.op.name]
+            clip_min = int(params.activation.attrs.a_min)
+            clip_max = int(params.activation.attrs.a_max)
+        else:
+            activation = "NONE"
+            clip_min = 0
+            clip_max = 0
+        scale_bias = vela_api.pack_biases(
+            biases=params.biases.tensor.data.asnumpy(),
+            ifm_scale=params.ifm.q_params.scale_f32,
+            ifm_dtype=np.dtype(params.ifm.dtype),
+            weight_scales=params.weights.q_params.scale_f32,
+            ofm_scale=params.ofm.q_params.scale_f32,
+            is_activation_tanh_or_sigmoid=activation in ["TANH", "SIGMOID"],
+        )
+        ethosu_conv2d = ethosu_ops.ethosu_conv2d(
+            ifm=post.args[0],
+            weight=relay.const(weights_values_ohwi, params.weights.values.dtype),
+            scale_bias=relay.const(scale_bias, "uint8"),
+            lut=relay.const([], dtype="int8"),
+            ifm_scale=float(params.ifm.q_params.scale_f32),
+            ifm_zero_point=int(params.ifm.q_params.zero_point),
+            weight_zero_point=int(params.weights.q_params.zero_point),
+            ofm_scale=float(params.ofm.q_params.scale_f32),
+            ofm_zero_point=int(params.ofm.q_params.zero_point),
+            kernel_shape=kernel_size_map[str(params.weights.layout)],
+            ofm_channels=params.ofm.shape[channels_map[str(params.ofm.layout)]],
+            strides=params.strides,
+            padding=params.padding,
+            dilation=params.dilation,
+            activation=activation,
+            clip_min=clip_min,
+            clip_max=clip_max,
+            upscale="NONE",
+            ifm_layout=str(params.ifm.layout),
+            ofm_layout=str(params.ofm.layout),
+        )
+        return ethosu_conv2d
+
+
+class LegalizeEthosU:
+    """This is the wrapper class to call graph-rewrites to perform graph transformation

Review comment:
       Ideally this should be "this is a pass which converts a Relay program into a form that can be accepted by the Ethos-U code generator.

##########
File path: python/tvm/relay/backend/contrib/ethosu/vela_api.py
##########
@@ -0,0 +1,314 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+conversions between TVM and Vela. Therefore, all interactions with the

Review comment:
       Nit: The first line is not a sentence worth explaining further. 

##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU supported operators."""
+import numpy as np
+
+from tvm.relay.expr import Constant
+from tvm.relay.op.contrib.register import register_pattern_table
+from tvm.relay.dataflow_pattern import wildcard, is_op, is_constant
+from tvm.relay.backend.contrib.ethosu.util import QConv2DArgs
+from tvm.relay.backend.contrib.ethosu.util import BiasAddArgs
+from tvm.relay.backend.contrib.ethosu.util import RequantArgs
+from tvm.relay.backend.contrib.ethosu.util import get_dim_value
+from ethosu.vela import api as vapi
+
+
+def check_strides(strides):
+    """Checks whether strides are within the limits supported by the hardware"""
+    stride_range = (1, 3)
+    smin, smax = stride_range
+    if not smax >= strides[0] >= smin:
+        return False
+    if not smax >= strides[1] >= smin:
+        return False
+    return True
+
+
+def check_valid_dtypes(tensor_params):
+    """Check whether dtypes are supported by the hardware"""
+    supported_dtypes = (np.uint8, np.int8)
+    for tep in tensor_params:
+        # Check for dtypes
+        if np.dtype(tep.dtype) not in supported_dtypes:
+            return False
+        # Check for shape sizes
+        if any(dimlen > 65536 for dimlen in tep.shape):
+            return False
+    return True
+
+
+def check_weights(weights, dilation):
+    """Checks whether weight tensor is compatible with HW"""
+    dilated_height_range = (1, 64)
+    dilated_hxw_range = (1, 64 * 64)
+    weights_limit = 127 * 65536
+    dilated_width = (weights.shape[get_dim_value(weights.layout, "W")] - 1) * dilation[0] + 1
+    dilated_height = (weights.shape[get_dim_value(weights.layout, "H")] - 1) * dilation[1] + 1
+    dh_min, dh_max = dilated_height_range
+    if not dh_min <= dilated_height <= dh_max:
+        return False
+    dilated_hxw = dilated_height * dilated_width
+    dhxw_min, dhxw_max = dilated_hxw_range
+    if not dhxw_min <= dilated_hxw <= dhxw_max:
+        return False
+    # A saturation upper bound check for accumulators
+    weights.values = weights.values - weights.q_params.zero_point
+    axis = (
+        get_dim_value(weights.layout, "H"),
+        get_dim_value(weights.layout, "W"),
+        get_dim_value(weights.layout, "I"),
+    )
+    sum_weights = np.amax(np.sum(np.absolute(weights.values), axis=axis))
+    if not sum_weights <= weights_limit:
+        return False
+    return True
+
+
+def check_bias(bias):
+    """Check whether the bias values fit in 40 bits"""
+    if bias and bias.dtype == np.dtype("int64"):
+        valid = all(len(bin(bias_value)[2:]) <= 40 for bias_value in bias.values)
+        return valid
+    return True
+
+
+def check_batch_size(ifm):
+    """Checks for the number of batches vela currently supports"""
+    if ifm.shape[0] != 1:
+        return False
+    return True
+
+
+def check_dilation(dilation):
+    """Checks whether dilation is within the limits supported by the hardware"""
+    dilation_range = (1, 2)
+    dmin, dmax = dilation_range
+    if not dmin <= dilation[0] <= dmax:
+        return False
+    if not dmin <= dilation[1] <= dmax:
+        return False
+    return True
+
+
+def check_padding(padding, bounds):
+    """Checks whether padding is within the limits supported by the hardware"""
+    if len(padding) != 4 or len(bounds) != 4:
+        return False
+    top, left, bottom, right = padding
+    topb, leftb, bottomb, rightb = bounds
+    if top > topb or left > leftb or bottom > bottomb or right > rightb:
+        return False
+    return True
+
+
+class TensorParams:
+    """
+    This class will parse a tvm Expr along with quantization scale
+    and zero point to populate parameters that are required
+    for the creation of tensors in Vela.
+    """
+
+    def __init__(self, tensor, layout=None, scale=None, zero_point=None):
+        self.tensor = tensor
+        if isinstance(tensor, Constant):
+            self.values = tensor.data.asnumpy()
+        else:
+            self.values = None
+        self.dtype = tensor.checked_type.dtype
+        self.shape = [int(i) for i in tensor.checked_type.shape]
+        self.layout = layout
+
+        if scale is not None and zero_point is not None:
+            self.q_params = vapi.NpuQuantization(
+                scale.data.asnumpy().astype("float32"), zero_point.data.asnumpy().astype(self.dtype)
+            )
+        else:
+            # put default values
+            self.q_params = vapi.NpuQuantization(1.0, 0)
+
+
+class QnnConv2DParams:
+    """
+    This class will parse a Call to a ethosu.qnn_conv2d_clip composite function
+    and extract quantization information of all the associated tensors.
+    """
+
+    composite_name = "ethosu.qnn_conv2d"
+    # The hardware only supports padding upto the numbers as follows
+    padding_bounds = [31, 31, 32, 32]
+    activation_map = {"clip": "CLIP"}
+
+    def __init__(self, func_body):
+        activation = None
+        if str(func_body.op) in self.activation_map.keys():
+            activation = func_body
+            requantize_op = activation.args[0]
+        else:
+            requantize_op = func_body
+        bias_add = requantize_op.args[0]
+        qnn_conv2d = bias_add.args[0]
+        data_layout = qnn_conv2d.attrs.data_layout
+        kernel_layout = qnn_conv2d.attrs.kernel_layout
+        # We consider the weights & biases as params as it should be a Constant
+        self.weights = TensorParams(
+            qnn_conv2d.args[QConv2DArgs.weights.value],
+            kernel_layout,
+            qnn_conv2d.args[QConv2DArgs.weights_scale.value],
+            qnn_conv2d.args[QConv2DArgs.weights_zero_point.value],
+        )
+
+        self.biases = TensorParams(
+            bias_add.args[BiasAddArgs.biases.value],
+            data_layout,
+            requantize_op.args[RequantArgs.ifm_scale.value],
+            requantize_op.args[RequantArgs.ifm_zero_point.value],
+        )
+        self.ifm = TensorParams(
+            qnn_conv2d.args[QConv2DArgs.ifm.value],
+            data_layout,
+            qnn_conv2d.args[QConv2DArgs.ifm_scale.value],
+            qnn_conv2d.args[QConv2DArgs.ifm_zero_point.value],
+        )
+        self.ofm = TensorParams(
+            func_body,
+            data_layout,
+            requantize_op.args[RequantArgs.ofm_scale.value],
+            requantize_op.args[RequantArgs.ofm_zero_point.value],
+        )
+        self.padding = qnn_conv2d.attrs.padding
+        self.strides = qnn_conv2d.attrs.strides
+        self.dilation = qnn_conv2d.attrs.dilation
+        self.activation = activation
+
+        # If groups are equal to channel, its a depthwise_conv2d
+        self.groups = qnn_conv2d.attrs.groups
+        self.is_depthwise = False
+        channels_axis = {"HWIO": 3, "HWOI": 2}
+        if qnn_conv2d.attrs.groups == self.weights.shape[channels_axis[kernel_layout]]:
+            self.is_depthwise = True
+
+    def is_valid(self):
+        """
+        Checks whether QnnConv2D with Clip has compatible attributes with HW
+        """
+        tensor_params = [self.weights, self.ifm, self.ofm]
+        if not check_valid_dtypes(tensor_params):
+            return False
+        if not check_weights(self.weights, self.dilation):
+            return False
+        if not check_bias(self.biases):
+            return False
+        if not check_strides(self.strides):
+            return False
+        if not check_batch_size(self.ifm):
+            return False
+        if not check_dilation(self.dilation):
+            return False
+        if not check_padding(self.padding, self.padding_bounds):
+            return False
+        legal_groups = [1, self.ofm.shape[3]]
+        if self.groups not in legal_groups:
+            return False
+        # This should be a valid QnnDepthwise2DParams, not QnnConv2DParams
+        if self.is_depthwise:
+            return False
+        return True
+
+
+def qnn_conv2d_pattern():
+    """
+    Create pattern for qnn.conv2D with optional fused relu

Review comment:
       Improve the grammar in these if possible. 

##########
File path: src/relay/backend/contrib/ethosu/preprocess.cc
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include <tvm/ir/error.h>
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/attrs/annotation.h>
+#include <tvm/relay/expr.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/transform.h>
+
+#include <unordered_map>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "../../../op/make_op.h"
+
+namespace tvm {
+namespace relay {
+namespace contrib {
+namespace ethosu {
+
+/*!
+ * \brief This expression rewriter will traverse the graph to find calls
+ * to all external functions. If they have multiple inputs and/or
+ * multiple outputs, the following has to be done :
+ * 1) If multiple inputs are present, they needed to be concat before the call.
+ * 2) Inside the external function they need to be split again to their original inputs.
+ * 3) If there are multiple outputs, they need to be concat at the end of external function.
+ * 4) Then, the concat output again need to be split and made the original tuple output in the
+ * main.
+ */
+class ExternalFuncIOHandler : public ExprRewriter {
+ public:
+  explicit ExternalFuncIOHandler(IRModule& module) : module_(module) {}
+  int count = 0;
+
+  Function InferType(const Function& expr, const IRModule& m) {
+    IRModule mod(m);
+    mod->Update(mod->GetGlobalVar("main"), expr);
+    mod = transform::InferType()(mod);
+    return Downcast<Function>(mod->Lookup("main"));
+  }
+
+  /*!
+   * \brief This function will take shape and compute
+   * the scalar size value for it to be use to create
+   * flat single dimensional tensors.
+   */
+  int64_t CalcSize(const Array<Integer>& shape) {
+    int size = 1;
+    for (auto dim_sz : shape) {
+      size = size * Downcast<Integer>(dim_sz)->value;
+    }
+    return size;
+  }
+
+  /*!
+   * \brief This will take a tensor and create a flattened
+   * tensor to be used by the concat.
+   */
+  Expr CreateFlattenTensor(const Expr& input) {
+    auto ishape = Downcast<Array<Integer>>(Downcast<TensorType>(input->checked_type())->shape);
+    int flatten_size = CalcSize(ishape);
+    Array<Integer> oshape = {Integer(flatten_size)};
+    return MakeReshape(input, oshape);
+  }
+
+  /*!
+   * \brief This will take flattened tensors and create
+   * a single concat'd tensor.
+   */
+  Expr CreateConcatTensor(const Array<Expr>& inputs) {
+    auto tuple = Tuple(inputs);
+    return MakeConcatenate(tuple, 0);
+  }
+
+  /*!
+   * \brief This will take a flattened concat'd tensor and use the original inputs shapes
+   * to recreate a Tuple of the original set of tensors.
+   */
+  Expr CreateSplitReshapedTensors(const Expr& input, const Array<Expr>& original_args) {
+    Array<Array<Integer>> shapes;
+    Array<Integer> flatten_tensor_sizes;
+    Array<IndexExpr> split_indices;
+    Array<Expr> rets;
+
+    int total_size = 0;
+    for (auto orig_arg : original_args) {
+      auto shape = Downcast<Array<Integer>>(Downcast<TensorType>(orig_arg->checked_type())->shape);
+      shapes.push_back(shape);
+      flatten_tensor_sizes.push_back(CalcSize(shape));
+      if (total_size != 0) {
+        split_indices.push_back(total_size);
+      }
+      total_size += CalcSize(shape);
+    }
+    auto split_outs = MakeSplit(input, split_indices, 0);
+    for (unsigned int i = 0; i < shapes.size(); i++) {
+      auto split_out = TupleGetItem(split_outs, i);
+      split_out->checked_type_ = original_args[i]->checked_type_;
+      rets.push_back(MakeReshape(split_out, shapes[i]));
+    }
+    return Tuple(rets);
+  }
+
+  /*!
+   * \brief Modify the external function to split the input as the original compute
+   * as required originally. Moreover, the outputs will be flattened and concat'd
+   * to make a single output. Finaly, the external function should only have a single input
+   * and a single output.
+   */
+  Function ModifyExternalFunction(const Function& func, GlobalVar gv, const DataType& dtype) {
+    Array<Expr> inputs;
+    Var ifms;
+    if (func->params.size() > 1) {
+      Array<Array<Integer>> shapes;
+      Array<Integer> flatten_tensor_sizes;
+      Array<IndexExpr> split_indices;
+
+      auto func_name = gv->name_hint;
+      int total_size = 0;
+      for (auto input : func->params) {
+        auto shape = Downcast<Array<Integer>>(Downcast<TensorType>(input->checked_type())->shape);
+        shapes.push_back(shape);
+        auto flat_size = CalcSize(shape);
+        flatten_tensor_sizes.push_back(flat_size);
+        if (total_size != 0) {
+          split_indices.push_back(total_size);
+        }
+        total_size += flat_size;
+      }
+      Array<PrimExpr> ifms_shape = {total_size};
+      ifms = Var(func_name + "_ifms", TensorType(ifms_shape, dtype));
+      auto split_outs = MakeSplit(ifms, split_indices, 0);
+      for (unsigned int i = 0; i < shapes.size(); i++) {
+        auto split_out = TupleGetItem(split_outs, i);
+        split_out->checked_type_ = func->params[i]->checked_type();
+        inputs.push_back(MakeReshape(split_out, shapes[i]));
+      }
+    } else {
+      CHECK_EQ(func->params.size(), 1);
+      inputs.push_back(func->params[0]);
+      ifms = func->params[0];
+    }
+    Map<Var, Expr> bind_map;
+    CHECK_EQ(func->params.size(), inputs.size());
+    for (size_t i = 0; i < inputs.size(); i++) {
+      bind_map.Set(func->params[i], inputs[i]);
+    }
+    auto core_compute_expr = Bind(func->body, bind_map);
+
+    // Creation of wrapper inside the external function
+    Array<Var> params = {ifms};
+    if (func->body->IsInstance<TupleNode>()) {
+      auto tuple_out = func->body.as<TupleNode>();
+      Array<Expr> reshaped_outputs;
+      for (unsigned int i = 0; i < tuple_out->fields.size(); i++) {
+        auto out = Downcast<Tuple>(core_compute_expr)->fields[i];
+        out->checked_type_ = tuple_out->fields[i]->checked_type_;
+        reshaped_outputs.push_back(CreateFlattenTensor(out));
+      }
+      auto concat_out = CreateConcatTensor(reshaped_outputs);
+      auto f = Function(params, concat_out, concat_out->checked_type_, {}, func->attrs);
+      return InferType(f, this->module_);
+    } else {
+      auto f =
+          Function(params, core_compute_expr, core_compute_expr->checked_type_, {}, func->attrs);
+      return InferType(f, this->module_);
+    }
+  }
+
+  Expr Rewrite_(const CallNode* call, const Expr& post) final {
+    auto post_call = Downcast<Call>(post);
+
+    if (auto glb_var_node = post_call->op.as<GlobalVarNode>()) {
+      auto glb_var = GetRef<GlobalVar>(glb_var_node);
+      auto func = Downcast<Function>(module_->functions[glb_var]);
+
+      // If the number of inputs and output are 1 --> no need to do anything
+      if (post_call->args.size() == 1 && !func->body->IsInstance<TupleNode>()) {
+        return post;
+      }
+      if (auto compiler = func->GetAttr<String>(attr::kCompiler)) {
+        if (compiler == "ethosu") {
+          auto ext_input = std::move(post_call->args[0]);
+          auto arg_dtype = Downcast<TensorType>(post_call->args[0]->checked_type())->dtype;
+          if (post_call->args.size() > 1) {
+            Array<Expr> reshaped_inputs;
+            for (const auto& arg : post_call->args) {
+              // All arguments should be of same data type
+              CHECK_EQ(arg_dtype, Downcast<TensorType>(arg->checked_type())->dtype)
+                  << "Currently NPU external functions require all inputs to be of same data "
+                     "type";
+              reshaped_inputs.push_back(CreateFlattenTensor(arg));
+            }
+            ext_input = CreateConcatTensor(reshaped_inputs);
+          }
+          auto ext_func = ModifyExternalFunction(func, glb_var, arg_dtype);
+          Array<Expr> new_args = {ext_input};
+          module_->Add(glb_var, ext_func);
+          Expr new_call = Call(glb_var, new_args);
+          if (func->body->IsInstance<TupleNode>()) {
+            auto orginal_tuple_out = Downcast<Tuple>(func->body);
+            new_call = CreateSplitReshapedTensors(new_call, orginal_tuple_out->fields);
+          }
+          return std::move(new_call);
+        }
+      }
+    }
+    return post;
+  }
+
+ private:
+  IRModule module_;
+};
+
+IRModule PreprocessExternalFuncIO_(IRModule module) {

Review comment:
       Passes shouldn't directly mutate the module contents, we assume that we copy if we want to mutate. 

##########
File path: python/tvm/relay/backend/contrib/__init__.py
##########
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""external backend codegen modules for relay."""

Review comment:
       Nit: can you capitalize this correctly? 

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The relay expression for split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+class EthosUConv2DRewriter(DFPatternCallback):
+    """Convert conv2d related composite functions to ethosu_conv2d operators"""
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.pattern = (wildcard().has_attr({"Composite": "ethosu.qnn_conv2d"}))(wildcard())
+
+    def callback(self, pre, post, node_map):
+        params = ethosu_patterns.QnnConv2DParams(post.op.body)
+        params.ifm.tensor = post.args[0]
+        channels_map = {
+            "NHWC": 3,
+        }
+        if str(params.ofm.layout) not in channels_map.keys():
+            raise UnsupportedLayout(str(params.ofm.layout))
+        kernel_size_map = {
+            "HWIO": params.weights.shape[0:2],
+            "OHWI": params.weights.shape[1:3],
+            "HWOI": params.weights.shape[0:2],
+        }
+        if str(params.weights.layout) not in kernel_size_map.keys():
+            raise UnsupportedLayout(str(params.weights.layout))
+        activation_map = {"clip": "CLIP"}
+        weight_to_ohwi_transform_map = {"HWIO": [3, 0, 1, 2]}
+        weights_values = params.weights.values
+        weights_values_ohwi = np.transpose(
+            weights_values, weight_to_ohwi_transform_map[str(params.weights.layout)]
+        )
+        if params.activation:
+            activation = activation_map[params.activation.op.name]
+            clip_min = int(params.activation.attrs.a_min)
+            clip_max = int(params.activation.attrs.a_max)
+        else:
+            activation = "NONE"
+            clip_min = 0
+            clip_max = 0
+        scale_bias = vela_api.pack_biases(
+            biases=params.biases.tensor.data.asnumpy(),
+            ifm_scale=params.ifm.q_params.scale_f32,
+            ifm_dtype=np.dtype(params.ifm.dtype),
+            weight_scales=params.weights.q_params.scale_f32,
+            ofm_scale=params.ofm.q_params.scale_f32,
+            is_activation_tanh_or_sigmoid=activation in ["TANH", "SIGMOID"],
+        )
+        ethosu_conv2d = ethosu_ops.ethosu_conv2d(
+            ifm=post.args[0],
+            weight=relay.const(weights_values_ohwi, params.weights.values.dtype),
+            scale_bias=relay.const(scale_bias, "uint8"),
+            lut=relay.const([], dtype="int8"),
+            ifm_scale=float(params.ifm.q_params.scale_f32),
+            ifm_zero_point=int(params.ifm.q_params.zero_point),
+            weight_zero_point=int(params.weights.q_params.zero_point),
+            ofm_scale=float(params.ofm.q_params.scale_f32),
+            ofm_zero_point=int(params.ofm.q_params.zero_point),
+            kernel_shape=kernel_size_map[str(params.weights.layout)],
+            ofm_channels=params.ofm.shape[channels_map[str(params.ofm.layout)]],
+            strides=params.strides,
+            padding=params.padding,
+            dilation=params.dilation,
+            activation=activation,
+            clip_min=clip_min,
+            clip_max=clip_max,
+            upscale="NONE",
+            ifm_layout=str(params.ifm.layout),
+            ofm_layout=str(params.ofm.layout),
+        )
+        return ethosu_conv2d
+
+
+class LegalizeEthosU:

Review comment:
       Is there a reason that these are just functions over functions? it would be better to convert these into passes if the intention is to convert an entire program. 

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr

Review comment:
       Is this the split operation (i.e. relay.Op or relay.Expr)?

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The relay expression for split operator
+
+        Returns
+        -------
+        section_begins : list

Review comment:
       What is a section begin?

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer

Review comment:
       ```suggestion
           """Currently, the split operator takes an array of indices or an integer
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are

Review comment:
       ```suggestion
       for the code generator. The code generator matches strided_slices which are
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported

Review comment:
       Can you expand this?

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The relay expression for split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+class EthosUConv2DRewriter(DFPatternCallback):
+    """Convert conv2d related composite functions to ethosu_conv2d operators"""

Review comment:
       ```suggestion
       """Convert conv2d related composite functions into ethosu_conv2d operators."""
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/errors.py
##########
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=super-init-not-called
+"""This module is to hold all type of errors associated Arm(R) Ethos(TM)-U NPU Codegen"""

Review comment:
       ```suggestion
   """This module defines all error types associated with the Arm(R) Ethos(TM)-U NPU code generator."""
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.

Review comment:
       Clarify this line.

##########
File path: python/tvm/relay/backend/contrib/ethosu/errors.py
##########
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=super-init-not-called
+"""This module is to hold all type of errors associated Arm(R) Ethos(TM)-U NPU Codegen"""
+
+
+class EthosUCodegenError(Exception):
+    """Base class for all exceptions related to Codegen"""
+
+    def __init__(self, data):
+        self.message = "EthosUCodegenError:" + data
+
+    def __str__(self):
+        return self.message
+
+
+class UnsupportedLayout(EthosUCodegenError):
+    """Raised when unsupported layout is encountered in the codegen"""

Review comment:
       ```suggestion
       """Raised when unsupported layout is encountered during code generation."""
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/te/dma.py
##########
@@ -0,0 +1,299 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unnecessary-lambda
+"""Tensor Expressions for operations supported by the DMA engine"""
+import tvm
+from tvm import te
+from tvm.topi.utils import equal_const_int
+
+
+def _pad_tensor(tensor, pad_before, pad_after=None):
+    """Generate a padded tensor.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to pad.
+    pad_before : tuple of int
+        The 'before' padding on each axis.
+    pad_after : tuple of int
+        The 'after' padding on each axis.
+    Returns
+    -------
+    _pad : callable
+        The padded tensor.
+
+    """
+    pad_after = pad_after or pad_before
+    dims = len(tensor.shape)
+    assert len(pad_before) == dims
+    assert len(pad_after) == dims
+
+    def _pad(*indices):
+        not_zero = []
+        index_tuple = []
+        for i in range(dims):
+            if equal_const_int(pad_before[i], 0) and equal_const_int(pad_after[i], 0):
+                index_tuple.append(indices[i])
+            else:
+                index_tuple.append(indices[i] - pad_before[i])
+                not_zero.append(indices[i] >= pad_before[i])
+                not_zero.append(indices[i] < tensor.shape[i] + pad_before[i])
+        if not_zero:
+            not_zero = tvm.tir.all(*not_zero)
+            return tvm.tir.if_then_else(not_zero, tensor(*index_tuple), tvm.tir.const(0, "uint8"))
+        return tensor(*index_tuple)
+
+    return _pad
+
+
+def read_compute(tensor, layout, zero_point, scale):
+    """A TE compute operator to represent a read.

Review comment:
       You can probably just say: "A tensor expression which represents a read." 

##########
File path: python/tvm/relay/backend/contrib/ethosu/preprocess.py
##########
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+"""Set of passes to pre-process the IRModule prior to codegen"""

Review comment:
       Can you clarify the scope of what this does?

##########
File path: python/tvm/relay/backend/contrib/ethosu/te/dma.py
##########
@@ -0,0 +1,299 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unnecessary-lambda
+"""Tensor Expressions for operations supported by the DMA engine"""

Review comment:
       ```suggestion
   """Tensor Expressions for operations supported by the Ethos-U DMA engine."""
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator

Review comment:
       Suggestion: move this commentary directly to the type or name them here so we can understand "which one" you are talking about. 

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen

Review comment:
       ```suggestion
   Helper utility Enums and Functions used through out code generation. 
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn conv2d arguments
+    """
+
+    ifm = 0
+    weights = 1
+    ifm_zero_point = 2
+    weights_zero_point = 3
+    ifm_scale = 4
+    weights_scale = 5
+
+
+class RequantArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn requantize arguments
+    """
+
+    ifm_scale = 1
+    ifm_zero_point = 2
+    ofm_scale = 3
+    ofm_zero_point = 4
+
+
+class BiasAddArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    biases = 1
+
+
+class ClipArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    a_min = 1
+    a_max = 2
+
+
+class MaxPoolArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm = 0
+
+
+class AddArgs(Enum):
+    """This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm0 = 0
+    ifm1 = 1
+    ifm0_scale = 2
+    ifm0_zero_point = 3
+    ifm1_scale = 4
+    ifm1_zero_point = 5
+    ofm_scale = 6
+    ofm_zero_point = 7
+
+
+def is_composite_func(func, name):
+    """
+    This a method to check whether the call is to

Review comment:
       ```suggestion
       This method checks whether the call is to
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn conv2d arguments
+    """
+
+    ifm = 0
+    weights = 1
+    ifm_zero_point = 2
+    weights_zero_point = 3
+    ifm_scale = 4
+    weights_scale = 5
+
+
+class RequantArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn requantize arguments
+    """
+
+    ifm_scale = 1
+    ifm_zero_point = 2
+    ofm_scale = 3
+    ofm_zero_point = 4
+
+
+class BiasAddArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    biases = 1
+
+
+class ClipArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    a_min = 1
+    a_max = 2
+
+
+class MaxPoolArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm = 0
+
+
+class AddArgs(Enum):
+    """This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm0 = 0
+    ifm1 = 1
+    ifm0_scale = 2
+    ifm0_zero_point = 3
+    ifm1_scale = 4
+    ifm1_zero_point = 5
+    ofm_scale = 6
+    ofm_zero_point = 7
+
+
+def is_composite_func(func, name):
+    """
+    This a method to check whether the call is to
+    a composite function of the "name".

Review comment:
       You should be able to use Pydoc to reference the argument in these cases.

##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU supported operators."""
+import numpy as np
+
+from tvm.relay.expr import Constant
+from tvm.relay.op.contrib.register import register_pattern_table
+from tvm.relay.dataflow_pattern import wildcard, is_op, is_constant
+from tvm.relay.backend.contrib.ethosu.util import QConv2DArgs
+from tvm.relay.backend.contrib.ethosu.util import BiasAddArgs
+from tvm.relay.backend.contrib.ethosu.util import RequantArgs
+from tvm.relay.backend.contrib.ethosu.util import get_dim_value
+from ethosu.vela import api as vapi
+
+
+def check_strides(strides):
+    """Checks whether strides are within the limits supported by the hardware"""

Review comment:
       Nit: might be worth expanding HW -> Ethos-U for better understanding when reading this without knowing exactly what file we are in. 

##########
File path: src/relay/backend/contrib/ethosu/preprocess.cc
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include <tvm/ir/error.h>
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/attrs/annotation.h>
+#include <tvm/relay/expr.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/transform.h>
+
+#include <unordered_map>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "../../../op/make_op.h"
+
+namespace tvm {
+namespace relay {
+namespace contrib {
+namespace ethosu {
+
+/*!
+ * \brief This expression rewriter will traverse the graph to find calls
+ * to all external functions. If they have multiple inputs and/or
+ * multiple outputs, the following has to be done :
+ * 1) If multiple inputs are present, they needed to be concat before the call.
+ * 2) Inside the external function they need to be split again to their original inputs.
+ * 3) If there are multiple outputs, they need to be concat at the end of external function.
+ * 4) Then, the concat output again need to be split and made the original tuple output in the
+ * main.
+ */
+class ExternalFuncIOHandler : public ExprRewriter {
+ public:
+  explicit ExternalFuncIOHandler(IRModule& module) : module_(module) {}
+  int count = 0;
+
+  Function InferType(const Function& expr, const IRModule& m) {
+    IRModule mod(m);
+    mod->Update(mod->GetGlobalVar("main"), expr);
+    mod = transform::InferType()(mod);
+    return Downcast<Function>(mod->Lookup("main"));
+  }
+
+  /*!
+   * \brief This function will take shape and compute
+   * the scalar size value for it to be use to create
+   * flat single dimensional tensors.
+   */
+  int64_t CalcSize(const Array<Integer>& shape) {
+    int size = 1;
+    for (auto dim_sz : shape) {
+      size = size * Downcast<Integer>(dim_sz)->value;
+    }
+    return size;
+  }
+
+  /*!
+   * \brief This will take a tensor and create a flattened
+   * tensor to be used by the concat.
+   */
+  Expr CreateFlattenTensor(const Expr& input) {
+    auto ishape = Downcast<Array<Integer>>(Downcast<TensorType>(input->checked_type())->shape);
+    int flatten_size = CalcSize(ishape);
+    Array<Integer> oshape = {Integer(flatten_size)};
+    return MakeReshape(input, oshape);
+  }
+
+  /*!
+   * \brief This will take flattened tensors and create
+   * a single concat'd tensor.
+   */
+  Expr CreateConcatTensor(const Array<Expr>& inputs) {
+    auto tuple = Tuple(inputs);
+    return MakeConcatenate(tuple, 0);
+  }
+
+  /*!
+   * \brief This will take a flattened concat'd tensor and use the original inputs shapes
+   * to recreate a Tuple of the original set of tensors.
+   */
+  Expr CreateSplitReshapedTensors(const Expr& input, const Array<Expr>& original_args) {
+    Array<Array<Integer>> shapes;
+    Array<Integer> flatten_tensor_sizes;
+    Array<IndexExpr> split_indices;
+    Array<Expr> rets;
+
+    int total_size = 0;
+    for (auto orig_arg : original_args) {
+      auto shape = Downcast<Array<Integer>>(Downcast<TensorType>(orig_arg->checked_type())->shape);
+      shapes.push_back(shape);
+      flatten_tensor_sizes.push_back(CalcSize(shape));
+      if (total_size != 0) {
+        split_indices.push_back(total_size);
+      }
+      total_size += CalcSize(shape);
+    }
+    auto split_outs = MakeSplit(input, split_indices, 0);
+    for (unsigned int i = 0; i < shapes.size(); i++) {
+      auto split_out = TupleGetItem(split_outs, i);
+      split_out->checked_type_ = original_args[i]->checked_type_;
+      rets.push_back(MakeReshape(split_out, shapes[i]));
+    }
+    return Tuple(rets);
+  }
+
+  /*!
+   * \brief Modify the external function to split the input as the original compute
+   * as required originally. Moreover, the outputs will be flattened and concat'd
+   * to make a single output. Finaly, the external function should only have a single input
+   * and a single output.
+   */
+  Function ModifyExternalFunction(const Function& func, GlobalVar gv, const DataType& dtype) {

Review comment:
       ```suggestion
     Function ModifyExternalFunction(const Function& func, const GlobalVar& gv, const DataType& dtype) {
   ```
   
   Can all of these be const?

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index

Review comment:
       I know I have left a lot of English nits, but would be good to make all of these full sentences with punctuation. 

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn conv2d arguments
+    """
+
+    ifm = 0

Review comment:
       Style: these are normally capitalized in Python I believe.

##########
File path: src/relay/op/contrib/ethosu/convolution.cc
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/op/contrib/ethosu/convolution.cc
+ * \brief Property def of the Arm(R) Ethos(TM)-U NPU convolution ops.

Review comment:
       Clarify this?
   ```suggestion
    * \brief Operator definitions for the Arm(R) Ethos(TM)-U NPU convolution ops.
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/vela_api.py
##########
@@ -0,0 +1,314 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+conversions between TVM and Vela. Therefore, all interactions with the
+Vela API are supposed to go through this adapter, with the hope that
+any changes to Vela API, TVM only needs to change this file.
+The following conversion APIs are added :
+    *Obtaining the best block config
+    *Compressing weights
+    *Packing biases
+"""
+import logging
+import math
+import numpy as np
+from ethosu.vela import api as vapi
+
+from tvm.relay.backend.contrib.ethosu import util
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("Ethos-U")
+
+VELA_TO_NP_DTYPES = {
+    vapi.NpuDataType.UINT8: np.uint8,
+    vapi.NpuDataType.UINT16: np.uint16,
+    vapi.NpuDataType.INT8: np.int8,
+    vapi.NpuDataType.INT16: np.int16,
+    vapi.NpuDataType.INT32: np.int32,
+}
+
+SCALE_BIAS_LENGTH = 10
+
+
+def get_optimal_block_config(npu_op, accel_type):
+    """
+    "The NPU's unit of work is known as a block. It will fetch block(s) from Input
+    Feature Map (IFM) and a compute block for Output Feature Map (OFM).
+    Therefore, we need to pick an optimal block configuration considering bandwidth
+    to bring IFM blocks and the number of OFM block computes need to happen
+    to cover the OFM as indicated by the npu op.
+
+    Parameters
+    ----------
+    npu_op : ethosu.vela.api.NpuOperation
+        The NPU operation and its params
+    accel_type : ethosu.vela.api.NpuAccelerator
+        The NPU accelerator variant
+    Returns
+    -------
+    ethosu.vela.api.NpuShape3d :
+        The optimal block config for the operator
+    """
+    all_valid_block_configs = vapi.npu_find_block_configs(npu_op, accel_type)
+    return _get_optimal_block_config(all_valid_block_configs)
+
+
+def _get_optimal_block_config(all_valid_block_configs):
+    """An internal function to get block config with largest depth
+    and then highest volume/area"""
+    assert isinstance(all_valid_block_configs, list)
+    for block_cfg in all_valid_block_configs:
+        assert isinstance(block_cfg, vapi.NpuShape3D)
+
+    # Getting the largest volume block for benchmarksing
+    all_valid_block_configs.sort(
+        key=lambda _cfg: _cfg.depth * _cfg.height * _cfg.width, reverse=True
+    )
+    largest_volume_block_config = all_valid_block_configs[0]
+    largest_volume = (
+        largest_volume_block_config.depth
+        * largest_volume_block_config.height
+        * largest_volume_block_config.width
+    )
+
+    all_valid_block_configs.sort(key=lambda _cfg: _cfg.depth, reverse=True)
+    max_d = all_valid_block_configs[0].depth
+    max_depth_block_configs = [_cfg for _cfg in all_valid_block_configs if _cfg.depth == max_d]
+    max_depth_block_configs.sort(key=lambda _cfg: _cfg.height * _cfg.width, reverse=True)
+    max_area = max_depth_block_configs[0].height * max_depth_block_configs[0].width
+    max_area_depth_block_configs = [
+        _cfg for _cfg in max_depth_block_configs if _cfg.height * _cfg.width == max_area
+    ]
+    # This to get a deterministic anwser everytime
+    max_area_depth_block_configs.sort(key=lambda _cfg: _cfg.height, reverse=True)
+    assert len(max_area_depth_block_configs) > 0
+    current_volume = (
+        max_area_depth_block_configs[0].depth
+        * max_area_depth_block_configs[0].height
+        * max_area_depth_block_configs[0].width
+    )
+    logger.info("Using block config=%s", max_area_depth_block_configs[0])
+    logger.info(
+        "Quality of the block config w.r.t. max volume block config=%s",
+        100.0 * (current_volume / largest_volume),
+    )
+    return max_area_depth_block_configs[0]
+
+
+def compress_weights(
+    weights,
+    weights_zp,
+    weights_layout,
+    ifm_bitdepth,
+    block_depth,
+    dilation,
+    accel_type,
+    is_depthwise=False,
+):
+    """Obtain compressed weights from vela

Review comment:
       Would be worth explaining more above, what does this mean to "obtain the compressed weights from vela"?

##########
File path: python/tvm/relay/backend/contrib/ethosu/te/dma.py
##########
@@ -0,0 +1,299 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unnecessary-lambda
+"""Tensor Expressions for operations supported by the DMA engine"""

Review comment:
       Can you maybe explain why all the operations need to be redefined here? I think it is due to the fact that Ethos-U code generator matches its own TE for lowering, but its hard to tell from the code.

##########
File path: python/tvm/relay/backend/contrib/ethosu/vela_api.py
##########
@@ -0,0 +1,314 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+conversions between TVM and Vela. Therefore, all interactions with the
+Vela API are supposed to go through this adapter, with the hope that
+any changes to Vela API, TVM only needs to change this file.
+The following conversion APIs are added :
+    *Obtaining the best block config
+    *Compressing weights
+    *Packing biases
+"""
+import logging
+import math
+import numpy as np
+from ethosu.vela import api as vapi
+
+from tvm.relay.backend.contrib.ethosu import util
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("Ethos-U")
+
+VELA_TO_NP_DTYPES = {
+    vapi.NpuDataType.UINT8: np.uint8,
+    vapi.NpuDataType.UINT16: np.uint16,
+    vapi.NpuDataType.INT8: np.int8,
+    vapi.NpuDataType.INT16: np.int16,
+    vapi.NpuDataType.INT32: np.int32,
+}
+
+SCALE_BIAS_LENGTH = 10
+
+
+def get_optimal_block_config(npu_op, accel_type):
+    """
+    "The NPU's unit of work is known as a block. It will fetch block(s) from Input
+    Feature Map (IFM) and a compute block for Output Feature Map (OFM).
+    Therefore, we need to pick an optimal block configuration considering bandwidth
+    to bring IFM blocks and the number of OFM block computes need to happen
+    to cover the OFM as indicated by the npu op.
+
+    Parameters
+    ----------
+    npu_op : ethosu.vela.api.NpuOperation
+        The NPU operation and its params
+    accel_type : ethosu.vela.api.NpuAccelerator
+        The NPU accelerator variant
+    Returns
+    -------
+    ethosu.vela.api.NpuShape3d :
+        The optimal block config for the operator
+    """
+    all_valid_block_configs = vapi.npu_find_block_configs(npu_op, accel_type)
+    return _get_optimal_block_config(all_valid_block_configs)
+
+
+def _get_optimal_block_config(all_valid_block_configs):
+    """An internal function to get block config with largest depth
+    and then highest volume/area"""
+    assert isinstance(all_valid_block_configs, list)
+    for block_cfg in all_valid_block_configs:
+        assert isinstance(block_cfg, vapi.NpuShape3D)
+
+    # Getting the largest volume block for benchmarksing
+    all_valid_block_configs.sort(
+        key=lambda _cfg: _cfg.depth * _cfg.height * _cfg.width, reverse=True
+    )
+    largest_volume_block_config = all_valid_block_configs[0]
+    largest_volume = (
+        largest_volume_block_config.depth
+        * largest_volume_block_config.height
+        * largest_volume_block_config.width
+    )
+
+    all_valid_block_configs.sort(key=lambda _cfg: _cfg.depth, reverse=True)
+    max_d = all_valid_block_configs[0].depth
+    max_depth_block_configs = [_cfg for _cfg in all_valid_block_configs if _cfg.depth == max_d]
+    max_depth_block_configs.sort(key=lambda _cfg: _cfg.height * _cfg.width, reverse=True)
+    max_area = max_depth_block_configs[0].height * max_depth_block_configs[0].width
+    max_area_depth_block_configs = [
+        _cfg for _cfg in max_depth_block_configs if _cfg.height * _cfg.width == max_area
+    ]
+    # This to get a deterministic anwser everytime
+    max_area_depth_block_configs.sort(key=lambda _cfg: _cfg.height, reverse=True)
+    assert len(max_area_depth_block_configs) > 0
+    current_volume = (
+        max_area_depth_block_configs[0].depth
+        * max_area_depth_block_configs[0].height
+        * max_area_depth_block_configs[0].width
+    )
+    logger.info("Using block config=%s", max_area_depth_block_configs[0])
+    logger.info(
+        "Quality of the block config w.r.t. max volume block config=%s",
+        100.0 * (current_volume / largest_volume),
+    )
+    return max_area_depth_block_configs[0]
+
+
+def compress_weights(
+    weights,
+    weights_zp,
+    weights_layout,
+    ifm_bitdepth,
+    block_depth,
+    dilation,
+    accel_type,
+    is_depthwise=False,
+):
+    """Obtain compressed weights from vela
+
+    Parameters
+    ----------
+    weights : numpy.ndarray
+        The raw weights
+    weights_zp : int
+        The zero point of the weights
+    weights_layout : str
+        A string literal indicating the layout
+        Supported values : HWIO, HWOI, OHWI
+    ifm_bitdepth : int
+        The bit depth of the ifm the weights are used with
+    block_depth : int
+        The depth of the optimal block config for the operator
+    dilation : tuple
+        A tuple of 2 elements indicating dilation in h and w
+    accel_type : ethosu.vela.api.NpuAccelerator
+        The NPU accelerator variant
+    is_depthwise : bool, Optional
+        This indicates whether the weights are compressed for depthwise convolution
+
+    Returns
+    -------
+    compressed_weights : bytearray
+        Compressed weights
+    """
+    layout_transform_indices = {"HWIO": (3, 0, 1, 2), "HWOI": (2, 0, 1, 3), "OHWI": (0, 1, 2, 3)}
+    assert weights_layout in layout_transform_indices.keys()
+    assert isinstance(weights_zp, np.int64)
+    weights = weights.astype(np.int64) - weights_zp
+    # Vela needs the weights in OHWI layout
+    weights_ohwi = np.transpose(weights, layout_transform_indices[weights_layout])
+    shape_ohwi = [
+        weights.shape[layout_transform_indices[weights_layout][0]],
+        weights.shape[layout_transform_indices[weights_layout][1]],
+        weights.shape[layout_transform_indices[weights_layout][2]],
+        weights.shape[layout_transform_indices[weights_layout][3]],
+    ]
+    block_traversal = calculate_block_traversal_mode(is_depthwise, shape_ohwi, ifm_bitdepth)
+    compressed_weights = vapi.npu_encode_weights(
+        accelerator=accel_type,
+        weights_volume=weights_ohwi,
+        dilation_xy=dilation,
+        ifm_bitdepth=ifm_bitdepth,
+        ofm_block_depth=block_depth,
+        is_depthwise=is_depthwise,
+        block_traversal=block_traversal,
+    )
+    return compressed_weights
+
+
+def calculate_block_traversal_mode(is_depthwise, weights_shape_ohwi, ifm_bitdepth):
+    """Calculate a block traversal mode given whether the op is depthwise convolution,
+    shape of weights and bit-depth of the ifm.
+    """
+
+    if is_depthwise:
+        return vapi.NpuBlockTraversal.DEPTH_FIRST
+    # Determine which block traversal strategy has better DPU utilization
+    kernel_size = weights_shape_ohwi[1] * weights_shape_ohwi[2]
+    depth_utilization = weights_shape_ohwi[3] / util.round_up(
+        weights_shape_ohwi[3], 32 if ifm_bitdepth == 8 else 16
+    )
+    part_kernel_utilization = (weights_shape_ohwi[3] / util.round_up(weights_shape_ohwi[3], 8)) * (
+        kernel_size / util.round_up(kernel_size, 4 if ifm_bitdepth == 8 else 2)
+    )
+    if part_kernel_utilization >= depth_utilization or weights_shape_ohwi[3] <= 8:
+        # Part-kernel first is always better for ifm depths <= 8
+        return vapi.NpuBlockTraversal.PART_KERNEL_FIRST
+    return vapi.NpuBlockTraversal.DEPTH_FIRST
+
+
+def pack_biases(
+    biases,
+    ifm_scale,
+    ifm_dtype,
+    weight_scales,
+    ofm_scale,
+    is_activation_tanh_or_sigmoid=False,
+):
+    """
+    Obtain packed bias bytearray as the hardware requires from

Review comment:
       Could use more explanation here as well. I know you guys are experts at this hardware but I have a feeling many of us will have to refactor/maintain this in the future. 

##########
File path: src/relay/backend/contrib/ethosu/preprocess.cc
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include <tvm/ir/error.h>
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/attrs/annotation.h>
+#include <tvm/relay/expr.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/transform.h>
+
+#include <unordered_map>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "../../../op/make_op.h"
+
+namespace tvm {
+namespace relay {
+namespace contrib {
+namespace ethosu {
+
+/*!
+ * \brief This expression rewriter will traverse the graph to find calls
+ * to all external functions. If they have multiple inputs and/or
+ * multiple outputs, the following has to be done :
+ * 1) If multiple inputs are present, they needed to be concat before the call.
+ * 2) Inside the external function they need to be split again to their original inputs.
+ * 3) If there are multiple outputs, they need to be concat at the end of external function.
+ * 4) Then, the concat output again need to be split and made the original tuple output in the
+ * main.
+ */
+class ExternalFuncIOHandler : public ExprRewriter {
+ public:
+  explicit ExternalFuncIOHandler(IRModule& module) : module_(module) {}
+  int count = 0;
+
+  Function InferType(const Function& expr, const IRModule& m) {
+    IRModule mod(m);
+    mod->Update(mod->GetGlobalVar("main"), expr);
+    mod = transform::InferType()(mod);
+    return Downcast<Function>(mod->Lookup("main"));
+  }
+
+  /*!
+   * \brief This function will take shape and compute
+   * the scalar size value for it to be use to create
+   * flat single dimensional tensors.
+   */
+  int64_t CalcSize(const Array<Integer>& shape) {
+    int size = 1;
+    for (auto dim_sz : shape) {
+      size = size * Downcast<Integer>(dim_sz)->value;
+    }
+    return size;
+  }
+
+  /*!
+   * \brief This will take a tensor and create a flattened
+   * tensor to be used by the concat.
+   */
+  Expr CreateFlattenTensor(const Expr& input) {
+    auto ishape = Downcast<Array<Integer>>(Downcast<TensorType>(input->checked_type())->shape);
+    int flatten_size = CalcSize(ishape);
+    Array<Integer> oshape = {Integer(flatten_size)};
+    return MakeReshape(input, oshape);
+  }
+
+  /*!
+   * \brief This will take flattened tensors and create
+   * a single concat'd tensor.
+   */
+  Expr CreateConcatTensor(const Array<Expr>& inputs) {
+    auto tuple = Tuple(inputs);
+    return MakeConcatenate(tuple, 0);
+  }
+
+  /*!
+   * \brief This will take a flattened concat'd tensor and use the original inputs shapes
+   * to recreate a Tuple of the original set of tensors.
+   */
+  Expr CreateSplitReshapedTensors(const Expr& input, const Array<Expr>& original_args) {
+    Array<Array<Integer>> shapes;
+    Array<Integer> flatten_tensor_sizes;
+    Array<IndexExpr> split_indices;
+    Array<Expr> rets;
+
+    int total_size = 0;
+    for (auto orig_arg : original_args) {
+      auto shape = Downcast<Array<Integer>>(Downcast<TensorType>(orig_arg->checked_type())->shape);
+      shapes.push_back(shape);
+      flatten_tensor_sizes.push_back(CalcSize(shape));
+      if (total_size != 0) {
+        split_indices.push_back(total_size);
+      }
+      total_size += CalcSize(shape);
+    }
+    auto split_outs = MakeSplit(input, split_indices, 0);
+    for (unsigned int i = 0; i < shapes.size(); i++) {
+      auto split_out = TupleGetItem(split_outs, i);
+      split_out->checked_type_ = original_args[i]->checked_type_;
+      rets.push_back(MakeReshape(split_out, shapes[i]));
+    }
+    return Tuple(rets);
+  }
+
+  /*!
+   * \brief Modify the external function to split the input as the original compute
+   * as required originally. Moreover, the outputs will be flattened and concat'd
+   * to make a single output. Finaly, the external function should only have a single input
+   * and a single output.
+   */
+  Function ModifyExternalFunction(const Function& func, GlobalVar gv, const DataType& dtype) {
+    Array<Expr> inputs;
+    Var ifms;
+    if (func->params.size() > 1) {
+      Array<Array<Integer>> shapes;
+      Array<Integer> flatten_tensor_sizes;
+      Array<IndexExpr> split_indices;
+
+      auto func_name = gv->name_hint;
+      int total_size = 0;
+      for (auto input : func->params) {
+        auto shape = Downcast<Array<Integer>>(Downcast<TensorType>(input->checked_type())->shape);
+        shapes.push_back(shape);
+        auto flat_size = CalcSize(shape);
+        flatten_tensor_sizes.push_back(flat_size);
+        if (total_size != 0) {
+          split_indices.push_back(total_size);
+        }
+        total_size += flat_size;
+      }
+      Array<PrimExpr> ifms_shape = {total_size};
+      ifms = Var(func_name + "_ifms", TensorType(ifms_shape, dtype));
+      auto split_outs = MakeSplit(ifms, split_indices, 0);
+      for (unsigned int i = 0; i < shapes.size(); i++) {
+        auto split_out = TupleGetItem(split_outs, i);
+        split_out->checked_type_ = func->params[i]->checked_type();
+        inputs.push_back(MakeReshape(split_out, shapes[i]));
+      }
+    } else {
+      CHECK_EQ(func->params.size(), 1);
+      inputs.push_back(func->params[0]);
+      ifms = func->params[0];
+    }
+    Map<Var, Expr> bind_map;
+    CHECK_EQ(func->params.size(), inputs.size());
+    for (size_t i = 0; i < inputs.size(); i++) {
+      bind_map.Set(func->params[i], inputs[i]);
+    }
+    auto core_compute_expr = Bind(func->body, bind_map);
+
+    // Creation of wrapper inside the external function
+    Array<Var> params = {ifms};

Review comment:
       @mbs-octoml can we ensure that these are duplicating the same kind of flattening code that we already have in the compiler? 

##########
File path: tests/python/contrib/test_ethosu/test_legalize.py
##########
@@ -0,0 +1,345 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument
+
+import pytest
+
+pytest.importorskip("ethosu.vela")
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.backend.contrib import ethosu
+from tvm.relay.backend.contrib.ethosu import legalize, preprocess
+from tvm.relay.dataflow_pattern import *
+from tvm.relay.op.contrib.ethosu import *
+import relay_ir_builder
+
+
+def test_split_indices_legalize():
+    def create_graph(axis):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_relu = relay.nn.relu(x)
+        split_o = relay.split(x_relu, [5, 20, 45], axis).tuple_value
+        return relay.Function([x], split_o)
+
+    def expected_mod_axis1():
+        expected_ir_string = """
+        #[version = "0.0.5"]

Review comment:
       You can use the text format to both do expected and input IR if you want, there are some examples of this elsewhere in the code base, just FYI. 

##########
File path: python/tvm/relay/backend/contrib/ethosu/vela_api.py
##########
@@ -0,0 +1,314 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+conversions between TVM and Vela. Therefore, all interactions with the
+Vela API are supposed to go through this adapter, with the hope that
+any changes to Vela API, TVM only needs to change this file.
+The following conversion APIs are added :
+    *Obtaining the best block config
+    *Compressing weights
+    *Packing biases
+"""
+import logging
+import math
+import numpy as np
+from ethosu.vela import api as vapi
+
+from tvm.relay.backend.contrib.ethosu import util
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("Ethos-U")
+
+VELA_TO_NP_DTYPES = {
+    vapi.NpuDataType.UINT8: np.uint8,
+    vapi.NpuDataType.UINT16: np.uint16,
+    vapi.NpuDataType.INT8: np.int8,
+    vapi.NpuDataType.INT16: np.int16,
+    vapi.NpuDataType.INT32: np.int32,
+}
+
+SCALE_BIAS_LENGTH = 10
+
+
+def get_optimal_block_config(npu_op, accel_type):
+    """
+    "The NPU's unit of work is known as a block. It will fetch block(s) from Input
+    Feature Map (IFM) and a compute block for Output Feature Map (OFM).
+    Therefore, we need to pick an optimal block configuration considering bandwidth
+    to bring IFM blocks and the number of OFM block computes need to happen
+    to cover the OFM as indicated by the npu op.
+
+    Parameters
+    ----------
+    npu_op : ethosu.vela.api.NpuOperation
+        The NPU operation and its params
+    accel_type : ethosu.vela.api.NpuAccelerator
+        The NPU accelerator variant
+    Returns
+    -------
+    ethosu.vela.api.NpuShape3d :
+        The optimal block config for the operator
+    """
+    all_valid_block_configs = vapi.npu_find_block_configs(npu_op, accel_type)
+    return _get_optimal_block_config(all_valid_block_configs)
+
+
+def _get_optimal_block_config(all_valid_block_configs):
+    """An internal function to get block config with largest depth
+    and then highest volume/area"""
+    assert isinstance(all_valid_block_configs, list)
+    for block_cfg in all_valid_block_configs:
+        assert isinstance(block_cfg, vapi.NpuShape3D)
+
+    # Getting the largest volume block for benchmarksing
+    all_valid_block_configs.sort(
+        key=lambda _cfg: _cfg.depth * _cfg.height * _cfg.width, reverse=True
+    )
+    largest_volume_block_config = all_valid_block_configs[0]
+    largest_volume = (
+        largest_volume_block_config.depth
+        * largest_volume_block_config.height
+        * largest_volume_block_config.width
+    )
+
+    all_valid_block_configs.sort(key=lambda _cfg: _cfg.depth, reverse=True)
+    max_d = all_valid_block_configs[0].depth
+    max_depth_block_configs = [_cfg for _cfg in all_valid_block_configs if _cfg.depth == max_d]
+    max_depth_block_configs.sort(key=lambda _cfg: _cfg.height * _cfg.width, reverse=True)
+    max_area = max_depth_block_configs[0].height * max_depth_block_configs[0].width
+    max_area_depth_block_configs = [
+        _cfg for _cfg in max_depth_block_configs if _cfg.height * _cfg.width == max_area
+    ]
+    # This to get a deterministic anwser everytime
+    max_area_depth_block_configs.sort(key=lambda _cfg: _cfg.height, reverse=True)
+    assert len(max_area_depth_block_configs) > 0
+    current_volume = (
+        max_area_depth_block_configs[0].depth
+        * max_area_depth_block_configs[0].height
+        * max_area_depth_block_configs[0].width
+    )
+    logger.info("Using block config=%s", max_area_depth_block_configs[0])
+    logger.info(
+        "Quality of the block config w.r.t. max volume block config=%s",
+        100.0 * (current_volume / largest_volume),
+    )
+    return max_area_depth_block_configs[0]
+
+
+def compress_weights(
+    weights,
+    weights_zp,
+    weights_layout,
+    ifm_bitdepth,
+    block_depth,
+    dilation,
+    accel_type,
+    is_depthwise=False,
+):
+    """Obtain compressed weights from vela
+
+    Parameters
+    ----------
+    weights : numpy.ndarray
+        The raw weights
+    weights_zp : int
+        The zero point of the weights
+    weights_layout : str
+        A string literal indicating the layout
+        Supported values : HWIO, HWOI, OHWI
+    ifm_bitdepth : int
+        The bit depth of the ifm the weights are used with
+    block_depth : int
+        The depth of the optimal block config for the operator
+    dilation : tuple
+        A tuple of 2 elements indicating dilation in h and w
+    accel_type : ethosu.vela.api.NpuAccelerator
+        The NPU accelerator variant
+    is_depthwise : bool, Optional
+        This indicates whether the weights are compressed for depthwise convolution
+
+    Returns
+    -------
+    compressed_weights : bytearray
+        Compressed weights
+    """
+    layout_transform_indices = {"HWIO": (3, 0, 1, 2), "HWOI": (2, 0, 1, 3), "OHWI": (0, 1, 2, 3)}
+    assert weights_layout in layout_transform_indices.keys()
+    assert isinstance(weights_zp, np.int64)
+    weights = weights.astype(np.int64) - weights_zp
+    # Vela needs the weights in OHWI layout
+    weights_ohwi = np.transpose(weights, layout_transform_indices[weights_layout])
+    shape_ohwi = [
+        weights.shape[layout_transform_indices[weights_layout][0]],
+        weights.shape[layout_transform_indices[weights_layout][1]],
+        weights.shape[layout_transform_indices[weights_layout][2]],
+        weights.shape[layout_transform_indices[weights_layout][3]],
+    ]
+    block_traversal = calculate_block_traversal_mode(is_depthwise, shape_ohwi, ifm_bitdepth)
+    compressed_weights = vapi.npu_encode_weights(
+        accelerator=accel_type,
+        weights_volume=weights_ohwi,
+        dilation_xy=dilation,
+        ifm_bitdepth=ifm_bitdepth,
+        ofm_block_depth=block_depth,
+        is_depthwise=is_depthwise,
+        block_traversal=block_traversal,
+    )
+    return compressed_weights
+
+
+def calculate_block_traversal_mode(is_depthwise, weights_shape_ohwi, ifm_bitdepth):
+    """Calculate a block traversal mode given whether the op is depthwise convolution,
+    shape of weights and bit-depth of the ifm.
+    """
+
+    if is_depthwise:
+        return vapi.NpuBlockTraversal.DEPTH_FIRST
+    # Determine which block traversal strategy has better DPU utilization
+    kernel_size = weights_shape_ohwi[1] * weights_shape_ohwi[2]
+    depth_utilization = weights_shape_ohwi[3] / util.round_up(
+        weights_shape_ohwi[3], 32 if ifm_bitdepth == 8 else 16
+    )
+    part_kernel_utilization = (weights_shape_ohwi[3] / util.round_up(weights_shape_ohwi[3], 8)) * (
+        kernel_size / util.round_up(kernel_size, 4 if ifm_bitdepth == 8 else 2)
+    )
+    if part_kernel_utilization >= depth_utilization or weights_shape_ohwi[3] <= 8:
+        # Part-kernel first is always better for ifm depths <= 8
+        return vapi.NpuBlockTraversal.PART_KERNEL_FIRST
+    return vapi.NpuBlockTraversal.DEPTH_FIRST
+
+
+def pack_biases(
+    biases,
+    ifm_scale,
+    ifm_dtype,
+    weight_scales,
+    ofm_scale,
+    is_activation_tanh_or_sigmoid=False,
+):
+    """
+    Obtain packed bias bytearray as the hardware requires from
+    Vela.
+    Parameters
+    ----------
+    biases : numpy.ndarray
+        The values of biases
+    ifm_scale : float
+        The quantization scale parameter of input feature map
+    ifm_dtype : numpy.dtype
+        The data type of input feature map data.
+    weight_scales : numpy.ndarray
+        The quantization scale parameter of weight feature map
+        This could be a tuple if per-channel quantization is present.
+    ofm_scale : float
+        The quantization scale parameter of output feature map.
+    is_activation_tanh_or_sigmoid : bool
+        Indicates whether the fused activation function is tanh or sigmoid.
+
+    Returns
+    -------
+    scale_bias : numpy.ndarray
+        Packed scales/biases as the hardware requires them.
+    """
+    # The BYOC infra should not partition anything else.
+    supported_ifm_dtypes = (np.uint8, np.int8, np.int16)
+    assert ifm_dtype in supported_ifm_dtypes
+
+    if weight_scales.size == 1:
+        weight_scales = [weight_scales] * biases.size
+
+    hw_bias_scales = _calculate_hw_bias_scales(
+        ifm_scale, weight_scales, ofm_scale, ifm_dtype, is_activation_tanh_or_sigmoid
+    )
+    assert len(hw_bias_scales) == biases.size
+    biases = biases.astype("int64")
+    packed_biases = bytearray()
+    for idx, scale in enumerate(hw_bias_scales):
+        packed_biases.extend(vapi.npu_encode_bias(biases[idx], *scale))
+    # Align to 16
+    # remainder = (len(packed_biases)) % 16
+    # if remainder > 0:
+    #     packed_biases.extend(bytearray(16 - remainder))
+    scale_bias = np.frombuffer(packed_biases, dtype=np.uint8)
+    scale_bias = np.reshape(scale_bias, (-1, 10))
+    return scale_bias
+
+
+def _quantize_scale(scale):
+    """Quantize floating point scale into 32-bit int scale with a 6-bit shift.
+    This is to be used with 8-bit data.
+    """
+    mantissa, exponent = math.frexp(scale)
+    mantissa_scaled = mantissa * (1 << 31)
+    mantissa_scaled = int(util.round_away_zero(mantissa_scaled))
+    required_shift = 31 - exponent
+    assert 0 <= required_shift < (1 << 6)
+    return mantissa_scaled, required_shift
+
+
+def _reduced_quantize_scale(scale):
+    """A reduction of precision is required for 16 bit data."""
+    mantissa_scaled, required_shift = _quantize_scale(scale)
+    # This is max a signed 16-bit number could represent
+    max_reduced_mantissa_scaled = (1 << 15) - 1
+    # if the current value is larger than pre-scaled max_reduced_mantissa_scaled
+    # we need to saturate the anwser to max_reduced_mantissa_scaled
+    if mantissa_scaled >= max_reduced_mantissa_scaled << 16:
+        reduced_mantissa_scaled = max_reduced_mantissa_scaled
+    else:
+        reduced_mantissa_scaled = (mantissa_scaled + (1 << 15)) >> 16
+    reduced_shift = required_shift - 16
+    return reduced_mantissa_scaled, reduced_shift
+
+
+def _calculate_hw_bias_scales(
+    ifm_scale, weight_scales, ofm_scale, ifm_dtype, is_faf_tanh_sigmoid=False
+):
+    """This function will produce a scale that is calculated using scales of ifm,
+    weights and ofm. It is also important to note that if per-channel / per-value
+    quantization required they should go into hw bias scales"""
+    if is_faf_tanh_sigmoid:
+        ifm_scale = ifm_scale * 0x3000
+    if ifm_dtype == np.uint8:
+        bias_scales = [np.double(ifm_scale * ws) / np.double(ofm_scale) for ws in weight_scales]
+    else:
+        assert ifm_dtype in (np.int8, np.int16)
+        ifm_scale_dbl = np.double(ifm_scale)
+        ofm_scale_dbl = np.double(ofm_scale)
+        bias_scales = [ifm_scale_dbl * np.double(ws) / ofm_scale_dbl for ws in weight_scales]
+
+    if ifm_dtype == np.int16:
+        hw_bias_scales = [_reduced_quantize_scale(bs) for bs in bias_scales]
+    else:
+        assert ifm_dtype in (np.uint8, np.int8)
+        hw_bias_scales = [_quantize_scale(bs) for bs in bias_scales]
+
+    return hw_bias_scales
+
+
+def get_target_accel_type():
+    """This is a helper function to convert cli accelerator type str argument

Review comment:
       I didn't quite understand this docstring either. 

##########
File path: tests/python/contrib/test_ethosu/relay_ir_builder.py
##########
@@ -0,0 +1,295 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Helper module to build relay operations for testing"""

Review comment:
       Is there a reason to not use normal IRBuilder for this? 

##########
File path: src/relay/op/contrib/ethosu/convolution.cc
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/op/contrib/ethosu/convolution.cc
+ * \brief Property def of the Arm(R) Ethos(TM)-U NPU convolution ops.
+ */
+#include "../../nn/convolution.h"
+
+#include <tvm/relay/base.h>
+#include <tvm/relay/op.h>
+#include <tvm/relay/qnn/attrs.h>
+#include <tvm/tir/analysis.h>
+#include <tvm/tir/data_layout.h>
+
+#include "../../../qnn/utils.h"
+#include "common.h"
+
+namespace tvm {
+namespace relay {
+namespace op {
+namespace contrib {
+namespace ethosu {
+
+/*! \brief Attributes used by the Ethos(TM)-U NPU convolution operator */
+struct EthosuConv2DAttrs : public tvm::AttrsNode<EthosuConv2DAttrs> {
+  double ifm_scale;
+  int ifm_zero_point;
+  int weight_zero_point;
+  double ofm_scale;
+  int ofm_zero_point;
+  Array<IndexExpr> kernel_shape;
+  IndexExpr ofm_channels;
+  Array<IndexExpr> strides;
+  Array<IndexExpr> padding;
+  Array<IndexExpr> dilation;
+  String activation;
+  int clip_min;
+  int clip_max;
+  String upscale;
+  tvm::String ifm_layout;
+  tvm::String ofm_layout;
+
+  TVM_DECLARE_ATTRS(EthosuConv2DAttrs, "relay.attrs.EthosuConv2DAttrs") {
+    TVM_ATTR_FIELD(ifm_scale).describe("The quantization scale for the Input Feature Map tensor.");
+    TVM_ATTR_FIELD(ifm_zero_point)
+        .describe("The quantization zero point for the Input Feature Map tensor.");
+    TVM_ATTR_FIELD(weight_zero_point)
+        .describe("The quantization zero point for the weight tensor.");
+    TVM_ATTR_FIELD(ofm_scale).describe("The quantization scale for the Output Feature Map tensor.");
+    TVM_ATTR_FIELD(ofm_zero_point)
+        .describe("The quantization zero point for the Output Feature Map tensor.");
+    TVM_ATTR_FIELD(kernel_shape)
+        .describe("The 2 dimensional kernel shape as (kernel_height, kernel_width).")
+        .set_default(NullValue<Array<IndexExpr> >());
+    TVM_ATTR_FIELD(ofm_channels)
+        .describe("The number of OFM channels.")
+        .set_default(NullValue<IndexExpr>());
+    TVM_ATTR_FIELD(strides)
+        .set_default(Array<IndexExpr>({1, 1}))
+        .describe("The 2 dimensional strides as (stride_height, stride_width).");
+    TVM_ATTR_FIELD(padding)
+        .set_default(Array<IndexExpr>({0, 0, 0, 0}))
+        .describe("The 4 dimensional padding as (pad_top, pad_left, pad_bottom, pad_right).");
+    TVM_ATTR_FIELD(dilation)
+        .set_default(Array<IndexExpr>({1, 1}))
+        .describe("The 2 dimensional dilation as (dilation_height, dilation_width).");
+    TVM_ATTR_FIELD(activation)
+        .describe(
+            "The activation function to use. "
+            "'NONE' - no activation function. "
+            "'CLIP' - clip the output between clip_min and clip_max. "
+            "'TANH' - tanh activation function. "
+            "'SIGMOID' - sigmoid activation function. "
+            "'LUT' - use a look-up table to perform the activation function.")
+        .set_default("NONE");
+    TVM_ATTR_FIELD(clip_min)
+        .describe("The minimum clipping value if activation = 'CLIP'.")
+        .set_default(0);
+    TVM_ATTR_FIELD(clip_max)
+        .describe("The maximum clipping value if activation = 'CLIP'.")
+        .set_default(0);
+    TVM_ATTR_FIELD(upscale)
+        .describe(
+            "The 2x2 upscaling mode to apply to the Input Feature Map tensor. "
+            "'NONE' - no upscaling. "
+            "'NEAREST' - upscale using nearest neighbour. "
+            "'ZEROS' - upscale using zeros.")
+        .set_default("NONE");
+    TVM_ATTR_FIELD(ifm_layout)
+        .set_default("NHWC")
+        .describe("The layout of the Input Feature Map tensor. Can be 'NHWC' or 'NHCWB16'.");
+    TVM_ATTR_FIELD(ofm_layout)
+        .set_default("NHWC")
+        .describe("The layout of the Output Feature Map tensor. Can be 'NHWC' or 'NHCWB16'.");
+  }
+};
+
+TVM_REGISTER_NODE_TYPE(EthosuConv2DAttrs);
+
+bool EthosuConv2DRel(const Array<Type>& types, int num_inputs, const Attrs& attrs,
+                     const TypeReporter& reporter) {
+  CHECK_EQ(types.size(), 5);
+  const auto* ifm = types[0].as<TensorTypeNode>();
+  const auto* weight = types[1].as<TensorTypeNode>();
+  const auto* scale_bias = types[2].as<TensorTypeNode>();
+  if (ifm == nullptr || weight == nullptr) return false;
+  const auto* param = attrs.as<EthosuConv2DAttrs>();
+  CHECK(param != nullptr) << "EthosuConv2DAttrs cannot be nullptr.";
+  CHECK(ifm->dtype == DataType::UInt(8) || ifm->dtype == DataType::Int(8))
+      << "Expected ethosu_conv2d type(uint8) or type(int8) for ifm but was " << ifm->dtype;
+  CHECK(weight->dtype == DataType::UInt(8) || weight->dtype == DataType::Int(8))
+      << "Expected ethosu_conv2d type(uint8) or type(int8) for weight but was " << weight->dtype;
+  CHECK(scale_bias->dtype == DataType::UInt(8))
+      << "Expected ethosu_conv2d type(uint8) for scale_bias but was " << scale_bias->dtype;
+
+  // The scale_bias should be provided as a tensor of size {ofm_channels, 10}
+  reporter->Assign(types[2], TensorType({weight->shape[0], 10}, DataType::UInt(8)));
+
+  // Assign weight type {ofm_channels, kernel_height, kernel_width, ifm_channels}
+  reporter->Assign(types[1], TensorType({param->ofm_channels, param->kernel_shape[0],
+                                         param->kernel_shape[1], weight->shape[3]},
+                                        weight->dtype));
+
+  // Assign ofm type
+  auto ofm_shape =
+      EthosuInferKernelOutput(ifm->shape, param->ifm_layout, param->ofm_layout, param->kernel_shape,
+                              param->ofm_channels, param->dilation, param->strides, param->padding);
+  reporter->Assign(types[4], TensorType(ofm_shape, ifm->dtype));
+  return true;
+}
+
+Expr MakeEthosuConv2D(Expr ifm, Expr weight, Expr scale_bias, Expr lut, double ifm_scale,
+                      int ifm_zero_point, int weight_zero_point, double ofm_scale,
+                      int ofm_zero_point, Array<IndexExpr> kernel_shape, IndexExpr ofm_channels,
+                      Array<IndexExpr> strides, Array<IndexExpr> padding, Array<IndexExpr> dilation,
+                      String activation, int clip_min, int clip_max, String upscale,
+                      String ifm_layout, String ofm_layout) {
+  auto attrs = make_object<EthosuConv2DAttrs>();
+  attrs->ifm_scale = ifm_scale;
+  attrs->ifm_zero_point = ifm_zero_point;
+  attrs->weight_zero_point = weight_zero_point;
+  attrs->ofm_scale = ofm_scale;
+  attrs->ofm_zero_point = ofm_zero_point;
+  attrs->kernel_shape = std::move(kernel_shape);
+  attrs->ofm_channels = std::move(ofm_channels);
+  attrs->strides = std::move(strides);
+  attrs->padding = std::move(padding);
+  attrs->dilation = std::move(dilation);
+  attrs->activation = std::move(activation);
+  attrs->clip_min = clip_min;
+  attrs->clip_max = clip_max;
+  attrs->upscale = std::move(upscale);
+  attrs->ifm_layout = std::move(ifm_layout);
+  attrs->ofm_layout = std::move(ofm_layout);
+  static const Op& op = Op::Get("contrib.ethosu.conv2d");
+  return Call(op, {ifm, weight, scale_bias, lut}, Attrs(attrs), {});
+}
+
+TVM_REGISTER_GLOBAL("relay.op._make.ethosu_conv2d").set_body_typed(MakeEthosuConv2D);
+
+RELAY_REGISTER_OP("contrib.ethosu.conv2d")
+    .describe(R"code(Arm(R) Ethos(TM)-U NPU 2D quantized convolution operator.
+
+This Relay operator corresponds to the hardware-implemented quantized

Review comment:
       This is a great example of a useful comment. 

##########
File path: tests/python/contrib/test_ethosu/relay_ir_builder.py
##########
@@ -0,0 +1,295 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Helper module to build relay operations for testing"""

Review comment:
       cc @mbs-octoml can you take another pass on testing code, want to try and consolidate 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695550561



##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn conv2d arguments
+    """
+
+    ifm = 0
+    weights = 1
+    ifm_zero_point = 2
+    weights_zero_point = 3
+    ifm_scale = 4
+    weights_scale = 5
+
+
+class RequantArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn requantize arguments
+    """
+
+    ifm_scale = 1
+    ifm_zero_point = 2
+    ofm_scale = 3
+    ofm_zero_point = 4
+
+
+class BiasAddArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    biases = 1
+
+
+class ClipArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    a_min = 1
+    a_max = 2
+
+
+class MaxPoolArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm = 0
+
+
+class AddArgs(Enum):
+    """This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm0 = 0
+    ifm1 = 1
+    ifm0_scale = 2
+    ifm0_zero_point = 3
+    ifm1_scale = 4
+    ifm1_zero_point = 5
+    ofm_scale = 6
+    ofm_zero_point = 7

Review comment:
       Yes, you are right, removed them.

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn conv2d arguments
+    """
+
+    ifm = 0
+    weights = 1
+    ifm_zero_point = 2
+    weights_zero_point = 3
+    ifm_scale = 4
+    weights_scale = 5
+
+
+class RequantArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn requantize arguments
+    """
+
+    ifm_scale = 1
+    ifm_zero_point = 2
+    ofm_scale = 3
+    ofm_zero_point = 4
+
+
+class BiasAddArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    biases = 1
+
+
+class ClipArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] ashutosh-arm commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
ashutosh-arm commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-914968687


   Yes, of course. What is the conflict btw?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] jroesch commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
jroesch commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-910741204


   @mbaret I think the API which computes unscheduled TE for a Relay Operator exists inside the guts of the `te_compiler.h` code. We should be able to expose it, can you clearly comment where you the exact API so Mark and I can look into replacing in parallel?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r701304104



##########
File path: src/relay/op/contrib/ethosu/convolution.cc
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/op/contrib/ethosu/convolution.cc
+ * \brief Property def of the Arm(R) Ethos(TM)-U NPU convolution ops.
+ */
+#include "../../nn/convolution.h"
+
+#include <tvm/relay/base.h>
+#include <tvm/relay/op.h>
+#include <tvm/relay/qnn/attrs.h>
+#include <tvm/tir/analysis.h>
+#include <tvm/tir/data_layout.h>
+
+#include "../../../qnn/utils.h"
+#include "common.h"
+
+namespace tvm {
+namespace relay {
+namespace op {
+namespace contrib {
+namespace ethosu {
+
+/*! \brief Attributes used by the Ethos(TM)-U NPU convolution operator */
+struct EthosuConv2DAttrs : public tvm::AttrsNode<EthosuConv2DAttrs> {
+  double ifm_scale;
+  int ifm_zero_point;
+  int weight_zero_point;
+  double ofm_scale;
+  int ofm_zero_point;
+  Array<IndexExpr> kernel_shape;
+  IndexExpr ofm_channels;
+  Array<IndexExpr> strides;
+  Array<IndexExpr> padding;
+  Array<IndexExpr> dilation;
+  String activation;
+  int clip_min;
+  int clip_max;
+  String upscale;
+  tvm::String ifm_layout;
+  tvm::String ofm_layout;

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695435831



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported

Review comment:
       ahh, HLCS is a concept of Vela that we dont use now.. should not have been here.

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbs-octoml commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-906697848


   Should the op defs go in python/tvm/relay/op/contrib/ethosu.py (where you currently have just the MergeComposite pattern defs), and the te defs should go in python/tvm/topi/ethosu/conv2d.py? We've gotten this far without a contrib under python/tvm/relay/backend so trying to follow existing convention as much as possible even with the custom scheduling would be nice.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbs-octoml commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-904969818


   Does this correspond to the proposed 'end-to-end conv2d' from the RFC? Are you cleared for showing any of the downstream work? I'm wondering if it would be better for me to checkout something further down the stream so I can better see where you're trying to go.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] Mousius commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
Mousius commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695768083



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,218 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel, no-value-for-parameter
+""" A set of passes to legalize some of operations for the NPU"""

Review comment:
       Remove leading space.

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,218 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel, no-value-for-parameter
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm import ir
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """This rewriting converts split operations into a sequence of
+    strided_slice operations, because codegen is going to be based
+    on strided_slices that will define the slice of the tensor that
+    will be fed to the consumer.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split operator takes an array of indices or an integer
+        indicating the number of splits. However, its an array of indices could
+        represent both cases, therefore this function just make it an array of
+        indices where each index represent the co-ordinate of beginning of each
+        section -- defines as section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The Relay Call expression for a split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))

Review comment:
       Can just return this without using another variable.

##########
File path: src/relay/backend/contrib/ethosu/preprocess.cc
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include <tvm/ir/error.h>
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/attrs/annotation.h>
+#include <tvm/relay/expr.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/transform.h>
+
+#include <unordered_map>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "../../../op/make_op.h"
+
+namespace tvm {
+namespace relay {
+namespace contrib {
+namespace ethosu {
+
+/*!
+ * \brief This expression rewriter will traverse the graph to find calls
+ * to all external functions. If they have multiple inputs and/or
+ * multiple outputs, the following has to be done :
+ * 1) If multiple inputs are present, they needed to be concat before the call.
+ * 2) Inside the external function they need to be split again to their original inputs.
+ * 3) If there are multiple outputs, they need to be concat at the end of external function.
+ * 4) Then, the concat output again need to be split and made the original tuple output in the
+ * main.
+ */
+class ExternalFuncIOHandler : public ExprRewriter {
+ public:
+  explicit ExternalFuncIOHandler(const IRModule& module) : module_(module) {}
+  int count = 0;
+
+  Function InferType(const Function& expr, const IRModule& m) {
+    IRModule mod(m);
+    mod->Update(mod->GetGlobalVar("main"), expr);
+    mod = transform::InferType()(mod);
+    return Downcast<Function>(mod->Lookup("main"));
+  }
+
+  /*!
+   * \brief This function will take shape and compute
+   * the scalar size value for it to be use to create
+   * flat single dimensional tensors.
+   */
+  int64_t CalcSize(const Array<Integer>& shape) {
+    int size = 1;
+    for (auto dim_sz : shape) {
+      size = size * Downcast<Integer>(dim_sz)->value;
+    }
+    return size;
+  }
+
+  /*!
+   * \brief This will take a tensor and create a flattened
+   * tensor to be used by the concat.
+   */
+  Expr CreateFlattenTensor(const Expr& input) {
+    auto ishape = Downcast<Array<Integer>>(Downcast<TensorType>(input->checked_type())->shape);
+    int flatten_size = CalcSize(ishape);
+    Array<Integer> oshape = {Integer(flatten_size)};
+    return MakeReshape(input, oshape);

Review comment:
       ```suggestion
       Array<Integer> output_shape = {Integer(flatten_size)};
       return MakeReshape(input, output_shape);
   ```

##########
File path: src/relay/op/contrib/ethosu/common.cc
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/op/contrib/ethosu/op_common.cc

Review comment:
       File name is wrong here.

##########
File path: tests/python/contrib/test_ethosu/test_preprocess.py
##########
@@ -0,0 +1,346 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument
+
+import pytest
+
+pytest.importorskip("ethosu.vela")
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+def set_func_attr(func, compile_name, symbol_name):
+    """
+    Helper function to attach attributes to the external function.
+    """
+    func = func.with_attr("Primitive", tvm.tir.IntImm("int32", 1))
+    func = func.with_attr("Inline", tvm.tir.IntImm("int32", 1))
+    func = func.with_attr("Compiler", compile_name)
+    func = func.with_attr("global_symbol", symbol_name)
+    return func
+
+
+def test_single_io():
+    """
+    This test will test the pass wont touch external functions that
+    have a single input and a single output.
+    """
+
+    def create_graph():
+        def create_external_func1(mod_, compiler_name, symbol_name):
+            x_int = relay.var("x_int", shape=(10, 10))
+            z0 = relay.nn.relu(x_int)
+            f1 = relay.Function([x_int], z0)
+            f1 = set_func_attr(f1, compiler_name, symbol_name)
+            glb_f1 = relay.GlobalVar(symbol_name)
+            mod_[glb_f1] = f1
+            mod_ = relay.transform.InferType()(mod_)
+            return glb_f1, mod_
+
+        mod = tvm.IRModule()
+        x = relay.var("x", shape=(10, 10))
+
+        glb_symbol_f1, mod = create_external_func1(mod, "ethosu", "ethosu_0")
+        r = relay.Call(glb_symbol_f1, [x])
+        main = relay.Function([x], r)
+        mod["main"] = main
+        mod = relay.transform.InferType()(mod)
+        return mod
+
+    mod = create_graph()
+    exp = create_graph()
+    mod = preprocess.preprocess_ext_io()(mod)
+    assert tvm.ir.structural_equal(mod, exp, map_free_vars=True)
+
+
+def test_2ins_single_out():
+    """
+    The test is check two inputs and a single output of external function
+    """
+
+    def create_graph():
+        def create_external_func1(mod_, compiler_name, symbol_name):
+            x_int = relay.var("x_int", shape=(10, 10))
+            w0_int = relay.var("w0_int", shape=(10, 10))
+            z0 = relay.add(x_int, w0_int)
+
+            f1 = relay.Function([x_int, w0_int], z0)
+            f1 = set_func_attr(f1, compiler_name, symbol_name)
+            glb_f1 = relay.GlobalVar(symbol_name)
+            mod_[glb_f1] = f1
+            mod_ = relay.transform.InferType()(mod_)
+            return glb_f1, mod_
+
+        mod = tvm.IRModule()
+
+        x = relay.var("x", shape=(10, 10))
+        w0 = relay.var("w0", shape=(10, 10))
+
+        glb_symbol_f1, mod = create_external_func1(mod, "ethosu", "ethosu_0")
+        r = relay.Call(glb_symbol_f1, [x, w0])
+        main = relay.Function([x, w0], r)
+        mod["main"] = main
+        mod = relay.transform.InferType()(mod)
+        return mod
+
+    def expected():
+        def create_external_func1(mod_, compiler_name, symbol_name):
+            ifms_int = relay.var("ifms_int", shape=[200])
+
+            # splits
+            (x_int_flat, w0_int_flat) = relay.split(ifms_int, [100])
+            # reshapes
+            x_int = relay.reshape(x_int_flat, newshape=(10, 10))
+            w0_int = relay.reshape(w0_int_flat, newshape=(10, 10))
+
+            z0 = relay.add(x_int, w0_int)
+            f1 = relay.Function([ifms_int], z0)
+            f1 = set_func_attr(f1, compiler_name, symbol_name)
+            glb_f1 = relay.GlobalVar(symbol_name)
+            mod_[glb_f1] = f1
+            mod_ = relay.transform.InferType()(mod_)
+            return glb_f1, mod_
+
+        mod = tvm.IRModule()
+
+        x = relay.var("x", shape=(10, 10))
+        w0 = relay.var("w0", shape=(10, 10))
+
+        # reshapes
+        x_reshaped = relay.reshape(x, newshape=100)
+        w0_reshaped = relay.reshape(w0, newshape=100)
+
+        # concat
+        ifms = relay.concatenate((x_reshaped, w0_reshaped), 0)
+
+        glb_symbol_f1, mod = create_external_func1(mod, "ethosu", "ethosu_0")
+        r = relay.Call(glb_symbol_f1, [ifms])
+        main = relay.Function([x, w0], r)
+        mod["main"] = main
+        mod = relay.transform.InferType()(mod)
+        return mod
+
+    mod = create_graph()
+    exp = expected()
+    mod = preprocess.preprocess_ext_io()(mod)
+    assert tvm.ir.structural_equal(mod, exp, map_free_vars=True)
+
+
+def test_single_in_2outs():
+    """
+    The test is to check a single input and two outputs of external function
+    """
+
+    def create_graph():
+        def create_external_func1(mod_, compiler_name, symbol_name):
+            x_int = relay.var("x_int", shape=(10, 10))
+
+            p0 = relay.nn.relu(x_int)
+            q0 = relay.tanh(x_int)
+            f1_o_tuple = relay.Tuple([p0, q0])
+
+            f1 = relay.Function([x_int], f1_o_tuple)
+            f1 = set_func_attr(f1, compiler_name, symbol_name)
+            glb_f1 = relay.GlobalVar(symbol_name)
+            mod_[glb_f1] = f1
+            mod_ = relay.transform.InferType()(mod_)
+            return glb_f1, mod_
+
+        mod = tvm.IRModule()
+        x = relay.var("x", shape=(10, 10))
+        glb_symbol_f1, mod = create_external_func1(mod, "ethosu", "ethosu_0")
+        pq_tuple = relay.Call(glb_symbol_f1, [x])
+        p0 = relay.TupleGetItem(pq_tuple, 0)
+        q0 = relay.TupleGetItem(pq_tuple, 1)
+        r = relay.concatenate((p0, q0), axis=0)
+        main = relay.Function([x], r)
+        mod["main"] = main
+        mod = relay.transform.InferType()(mod)
+        return mod
+
+    def expected():
+        def create_external_func1(mod_, compiler_name, symbol_name):
+            x_int = relay.var("x_int", shape=(10, 10))
+
+            p0 = relay.nn.relu(x_int)
+            q0 = relay.tanh(x_int)
+
+            # reshapes
+            p0_reshaped = relay.reshape(p0, newshape=100)
+            q0_reshaped = relay.reshape(q0, newshape=100)
+            ofms = relay.concatenate((p0_reshaped, q0_reshaped), 0)
+
+            f1 = relay.Function([x_int], ofms)
+            f1 = set_func_attr(f1, compiler_name, symbol_name)
+            glb_f1 = relay.GlobalVar(symbol_name)
+            mod_[glb_f1] = f1
+            mod_ = relay.transform.InferType()(mod_)
+            return glb_f1, mod_
+
+        mod = tvm.IRModule()
+        x = relay.var("x", shape=(10, 10))
+        glb_symbol_f1, mod = create_external_func1(mod, "ethosu", "ethosu_0")
+        ofms = relay.Call(glb_symbol_f1, [x])
+
+        # splits
+        (p0_flat, q0_flat) = relay.split(ofms, [100])
+        # reshapes
+        p0_flat_reshaped = relay.reshape(p0_flat, newshape=(10, 10))
+        q0_flat_reshaped = relay.reshape(q0_flat, newshape=(10, 10))
+        # original output
+        tuple_out = relay.Tuple([p0_flat_reshaped, q0_flat_reshaped])
+
+        p0 = relay.TupleGetItem(tuple_out, 0)
+        q0 = relay.TupleGetItem(tuple_out, 1)
+        r = relay.concatenate((p0, q0), axis=0)
+        main = relay.Function([x], r)
+        mod["main"] = main
+        mod = relay.transform.InferType()(mod)
+        return mod
+
+    mod = create_graph()
+    exp = expected()
+    mod = relay.transform.InferType()(mod)
+    mod = preprocess.preprocess_ext_io()(mod)
+    assert tvm.ir.structural_equal(mod, exp, map_free_vars=True)
+
+
+def test_4ins_2outs():
+    """
+    The test is to check a 4 inputs and two outputs of external function.
+    This just stand as a general test for multiple ins/outs.
+    """
+
+    def create_graph():
+        def create_external_func1(mod_, compiler_name, symbol_name):
+            x_int = relay.var("x_int", shape=(10, 10))
+            w0_int = relay.var("w0_int", shape=(10, 10))
+            w1_int = relay.var("w1_int", shape=(10, 10))
+            w2_int = relay.var("w2_int", shape=(10, 10))
+
+            z0 = relay.add(x_int, w0_int)
+            p0 = relay.subtract(z0, w1_int)
+            q0 = relay.multiply(z0, w2_int)
+            f1_o_tuple = relay.Tuple([p0, q0])
+
+            f1 = relay.Function([x_int, w0_int, w1_int, w2_int], f1_o_tuple)
+            f1 = set_func_attr(f1, compiler_name, symbol_name)
+            glb_f1 = relay.GlobalVar(symbol_name)
+            mod_[glb_f1] = f1
+            mod_ = relay.transform.InferType()(mod_)
+            return glb_f1, mod_
+
+        mod = tvm.IRModule()
+
+        x = relay.var("x", shape=(10, 10))
+        w0 = relay.var("w0", shape=(10, 10))
+        w1 = relay.var("w1", shape=(10, 10))
+        w2 = relay.var("w2", shape=(10, 10))
+
+        glb_symbol_f1, mod = create_external_func1(mod, "ethosu", "ethosu_0")
+        pq_tuple = relay.Call(glb_symbol_f1, [x, w0, w1, w2])
+
+        p0 = relay.TupleGetItem(pq_tuple, 0)
+        q0 = relay.TupleGetItem(pq_tuple, 1)
+        r = relay.concatenate((p0, q0), axis=0)
+        main = relay.Function([x, w0, w1, w2], r)
+        mod["main"] = main
+        mod = relay.transform.InferType()(mod)
+        return mod
+
+    def expected():
+        def create_external_func1(mod_, compiler_name, symbol_name):
+            ifms_int = relay.var("ifms_int", shape=[400])
+
+            # splits
+            (x_int_flat, w0_int_flat, w1_int_flat, w2_int_flat) = relay.split(
+                ifms_int, [100, 200, 300]
+            )
+            # reshapes
+            x_int = relay.reshape(x_int_flat, newshape=(10, 10))
+            w0_int = relay.reshape(w0_int_flat, newshape=(10, 10))
+            w1_int = relay.reshape(w1_int_flat, newshape=(10, 10))
+            w2_int = relay.reshape(w2_int_flat, newshape=(10, 10))
+
+            z0 = relay.add(x_int, w0_int)
+            p0 = relay.subtract(z0, w1_int)
+            q0 = relay.multiply(z0, w2_int)
+            # f1_o_tuple = relay.Tuple([p0, q0])
+
+            # reshapes
+            p0_reshaped = relay.reshape(p0, newshape=100)
+            q0_reshaped = relay.reshape(q0, newshape=100)
+            ofms = relay.concatenate((p0_reshaped, q0_reshaped), 0)
+
+            f1 = relay.Function([ifms_int], ofms)
+            f1 = set_func_attr(f1, compiler_name, symbol_name)
+            glb_f1 = relay.GlobalVar(symbol_name)
+            mod_[glb_f1] = f1
+            mod_ = relay.transform.InferType()(mod_)
+            return glb_f1, mod_
+
+        mod = tvm.IRModule()
+
+        x = relay.var("x", shape=(10, 10))
+        w0 = relay.var("w0", shape=(10, 10))
+        w1 = relay.var("w1", shape=(10, 10))
+        w2 = relay.var("w2", shape=(10, 10))
+
+        # reshapes
+        x_reshaped = relay.reshape(x, newshape=100)
+        w0_reshaped = relay.reshape(w0, newshape=100)
+        w1_reshaped = relay.reshape(w1, newshape=100)
+        w2_reshaped = relay.reshape(w2, newshape=100)
+
+        # concat
+        ifms = relay.concatenate((x_reshaped, w0_reshaped, w1_reshaped, w2_reshaped), 0)
+
+        # call
+        glb_func, mod = create_external_func1(mod, "ethosu", "ethosu_0")
+        ofms = relay.Call(glb_func, [ifms])
+
+        # splits
+        (p0_flat, q0_flat) = relay.split(ofms, [100])
+        # reshapes
+        p0_flat_reshaped = relay.reshape(p0_flat, newshape=(10, 10))
+        q0_flat_reshaped = relay.reshape(q0_flat, newshape=(10, 10))
+        # original output
+        tuple_out = relay.Tuple([p0_flat_reshaped, q0_flat_reshaped])
+
+        p0 = relay.TupleGetItem(tuple_out, 0)
+        q0 = relay.TupleGetItem(tuple_out, 1)
+
+        r = relay.concatenate((p0, q0), axis=0)
+        main = relay.Function([x, w0, w1, w2], r)
+        mod["main"] = main
+        mod = relay.transform.InferType()(mod)
+        return mod
+
+    mod = create_graph()
+    exp = expected()
+    mod = preprocess.preprocess_ext_io()(mod)
+    assert tvm.ir.structural_equal(mod, exp, map_free_vars=True)
+
+
+if __name__ == "__main__":
+    test_2ins_single_out()
+    test_single_io()
+    test_4ins_2outs()
+    test_single_in_2outs()

Review comment:
       ```suggestion
       sys.exit(pytest.main([__file__] + sys.argv[1:]))
   ```

##########
File path: tests/python/contrib/test_ethosu/test_legalize.py
##########
@@ -0,0 +1,333 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument
+
+import pytest
+
+pytest.importorskip("ethosu.vela")
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.backend.contrib import ethosu
+from tvm.relay.backend.contrib.ethosu import legalize, preprocess
+from tvm.relay.dataflow_pattern import *
+from tvm.relay.op.contrib.ethosu import *
+import relay_ir_builder
+
+
+def test_split_indices_legalize():
+    def create_graph(axis):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_relu = relay.nn.relu(x)
+        split_o = relay.split(x_relu, [5, 20, 45], axis).tuple_value

Review comment:
       `split_o` ?

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,218 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel, no-value-for-parameter
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm import ir
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """This rewriting converts split operations into a sequence of
+    strided_slice operations, because codegen is going to be based
+    on strided_slices that will define the slice of the tensor that
+    will be fed to the consumer.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split operator takes an array of indices or an integer
+        indicating the number of splits. However, its an array of indices could
+        represent both cases, therefore this function just make it an array of
+        indices where each index represent the co-ordinate of beginning of each
+        section -- defines as section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The Relay Call expression for a split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+@ir.transform.module_pass(opt_level=1)
+class LegalizeSplit:
+    """This is the pass that wraps SplitRewriter"""
+
+    def transform_module(self, mod, ctx):
+        for gv, func in mod.functions.items():

Review comment:
       `gv` ?

##########
File path: src/relay/op/contrib/ethosu/common.cc
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/op/contrib/ethosu/op_common.cc
+ * \brief A set of utilities and common functionality for Arm(R) Ethos(TM)-U NPU QNN ops.
+ */
+
+#include "common.h"
+
+#include "../../op_common.h"
+
+namespace tvm {
+namespace relay {
+namespace op {
+namespace contrib {
+namespace ethosu {
+
+Array<IndexExpr> EthosuInferKernelOutput(Array<IndexExpr> ifm_shape, String ifm_layout,
+                                         String ofm_layout, Array<IndexExpr> kernel_shape,
+                                         IndexExpr ofm_channels, Array<IndexExpr> dilation,
+                                         Array<IndexExpr> strides, Array<IndexExpr> padding) {
+  // In the case of NHCWB16, convert the ifm shape to NHW (C not required for this function)
+  if (ifm_layout == "NHCWB16") {
+    ifm_shape = {ifm_shape[0], ifm_shape[1], ifm_shape[3]};
+  }
+  Array<IndexExpr> oshape({ifm_shape[0], 0, 0, ofm_channels});
+
+  IndexExpr dilated_ksize_y = 1 + (kernel_shape[0] - 1) * dilation[0];
+  IndexExpr dilated_ksize_x = 1 + (kernel_shape[1] - 1) * dilation[1];
+  IndexExpr pad_h, pad_w;
+  GetPaddingHeightWidth(padding, &pad_h, &pad_w);
+  oshape.Set(1, indexdiv(ifm_shape[1] + pad_h - dilated_ksize_y, strides[0]) + 1);

Review comment:
       `output_shape`

##########
File path: python/tvm/relay/backend/contrib/ethosu/errors.py
##########
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=super-init-not-called
+"""This module defines all error types associated with the Arm(R) Ethos(TM)-U NPU code generator."""
+
+
+class EthosUCodegenError(Exception):
+    """Base class for all exceptions related to code generation"""
+
+    def __init__(self, data):
+        self.message = "EthosUCodegenError:" + data
+
+    def __str__(self):
+        return self.message
+
+
+class UnsupportedLayout(EthosUCodegenError):
+    """Raised when unsupported layout is encountered during code generation."""
+
+    def __init__(self, layout):
+        super().__init__(f"Unsupported Layout {layout}")
+
+    def __str__(self):

Review comment:
       I don't think you need to redefine `__str__`.

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,183 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out code generation.
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enum to obtain the correct index
+    of qnn.conv2d arguments.
+    """
+
+    IFM = 0
+    WEIGHTS = 1
+    IFM_ZERO_POINT = 2
+    WEIGHTS_ZERO_POINT = 3
+    IFM_SCALE = 4
+    WEIGHTS_SCALE = 5
+
+
+class RequantArgs(Enum):
+    """
+    This is a helper enum to obtain the correct index
+    of qnn.requantize arguments.
+    """
+
+    IFM_SCALE = 1
+    IFM_ZERO_POINT = 2
+    OFM_SCALE = 3
+    OFM_ZERO_POINT = 4
+
+
+class BiasAddArgs(Enum):
+    """
+    This is a helper enums to obtain the correct index
+    of qnn.bias_add arguments.
+    """
+
+    BIASES = 1
+
+
+class ClipArgs(Enum):
+    """
+    This is a helper enums to obtain the correct index
+    of clip arguments.
+    """
+
+    A_MIN = 1
+    A_MAX = 2
+
+
+def is_composite_func(func, name):
+    """
+    This method checks whether the call is to
+    a composite function of a given name.
+
+    Parameters
+    ----------
+    func : relay.Function
+        The header to be displayed along with the dump.
+
+    name : str
+        The candidate name to be checked
+
+    Returns
+    --------
+    a boolean
+    """
+
+    if not hasattr(func, "attrs"):
+        return False
+    if "Composite" not in func.attrs.keys():
+        return False
+    composite_name = func.attrs["Composite"]
+
+    if composite_name != name:
+        return False
+    return True

Review comment:
       ```suggestion
       return composite_name == name
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,218 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel, no-value-for-parameter
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm import ir
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """This rewriting converts split operations into a sequence of
+    strided_slice operations, because codegen is going to be based
+    on strided_slices that will define the slice of the tensor that
+    will be fed to the consumer.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split operator takes an array of indices or an integer
+        indicating the number of splits. However, its an array of indices could
+        represent both cases, therefore this function just make it an array of
+        indices where each index represent the co-ordinate of beginning of each
+        section -- defines as section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The Relay Call expression for a split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+@ir.transform.module_pass(opt_level=1)
+class LegalizeSplit:
+    """This is the pass that wraps SplitRewriter"""
+
+    def transform_module(self, mod, ctx):
+        for gv, func in mod.functions.items():
+            func = rewrite(SplitRewriter(), func)
+            mod.update_func(gv, func)
+        return mod
+
+    def __call__(self, *args, **kwargs):
+        pass
+
+
+class EthosUConv2DRewriter(DFPatternCallback):
+    """Convert conv2d related composite functions into ethosu_conv2d operators"""
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.pattern = (wildcard().has_attr({"Composite": "ethosu.qnn_conv2d"}))(wildcard())
+
+    def callback(self, pre, post, node_map):
+        params = ethosu_patterns.QnnConv2DParams(post.op.body)
+        params.ifm.tensor = post.args[0]
+        channels_map = {
+            "NHWC": 3,
+        }
+        if str(params.ofm.layout) not in channels_map.keys():
+            raise UnsupportedLayout(str(params.ofm.layout))
+        kernel_size_map = {
+            "HWIO": params.weights.shape[0:2],
+            "OHWI": params.weights.shape[1:3],
+            "HWOI": params.weights.shape[0:2],
+        }
+        if str(params.weights.layout) not in kernel_size_map.keys():
+            raise UnsupportedLayout(str(params.weights.layout))
+        activation_map = {"clip": "CLIP"}
+        weight_to_ohwi_transform_map = {"HWIO": [3, 0, 1, 2]}
+        weights_values = params.weights.values
+        weights_values_ohwi = np.transpose(
+            weights_values, weight_to_ohwi_transform_map[str(params.weights.layout)]
+        )
+        if params.activation:
+            activation = activation_map[params.activation.op.name]
+            clip_min = int(params.activation.attrs.a_min)
+            clip_max = int(params.activation.attrs.a_max)
+        else:
+            activation = "NONE"
+            clip_min = 0
+            clip_max = 0
+        scale_bias = vela_api.pack_biases(
+            biases=params.biases.tensor.data.asnumpy(),
+            ifm_scale=params.ifm.q_params.scale_f32,
+            ifm_dtype=np.dtype(params.ifm.dtype),
+            weight_scales=params.weights.q_params.scale_f32,
+            ofm_scale=params.ofm.q_params.scale_f32,
+            is_activation_tanh_or_sigmoid=activation in ["TANH", "SIGMOID"],
+        )
+        ethosu_conv2d = ethosu_ops.ethosu_conv2d(
+            ifm=post.args[0],
+            weight=relay.const(weights_values_ohwi, params.weights.values.dtype),
+            scale_bias=relay.const(scale_bias, "uint8"),
+            lut=relay.const([], dtype="int8"),
+            ifm_scale=float(params.ifm.q_params.scale_f32),
+            ifm_zero_point=int(params.ifm.q_params.zero_point),
+            weight_zero_point=int(params.weights.q_params.zero_point),
+            ofm_scale=float(params.ofm.q_params.scale_f32),
+            ofm_zero_point=int(params.ofm.q_params.zero_point),
+            kernel_shape=kernel_size_map[str(params.weights.layout)],
+            ofm_channels=params.ofm.shape[channels_map[str(params.ofm.layout)]],
+            strides=params.strides,
+            padding=params.padding,
+            dilation=params.dilation,
+            activation=activation,
+            clip_min=clip_min,
+            clip_max=clip_max,
+            upscale="NONE",
+            ifm_layout=str(params.ifm.layout),
+            ofm_layout=str(params.ofm.layout),
+        )
+        return ethosu_conv2d
+
+
+@ir.transform.module_pass(opt_level=1)
+class LegalizeEthosUConv2D:
+    """This is the pass that wraps the EthosUConv2DRewriter"""
+
+    def transform_module(self, mod, ctx):
+        for gv, func in mod.functions.items():
+            func = rewrite(EthosUConv2DRewriter(), func)
+            mod.update_func(gv, func)
+        return mod
+
+    def __call__(self, *args, **kwargs):

Review comment:
       Same here?

##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU supported operators."""
+import numpy as np
+
+from tvm.relay.expr import Constant
+from tvm.relay.op.contrib.register import register_pattern_table
+from tvm.relay.dataflow_pattern import wildcard, is_op, is_constant
+from tvm.relay.backend.contrib.ethosu.util import QConv2DArgs
+from tvm.relay.backend.contrib.ethosu.util import BiasAddArgs
+from tvm.relay.backend.contrib.ethosu.util import RequantArgs
+from tvm.relay.backend.contrib.ethosu.util import get_dim_value
+from ethosu.vela import api as vapi
+
+
+def check_strides(strides):
+    """This function checks whether strides are within the limits supported by the NPU"""
+    stride_range = (1, 3)
+    smin, smax = stride_range
+    if not smax >= strides[0] >= smin:
+        return False
+    if not smax >= strides[1] >= smin:
+        return False
+    return True
+
+
+def check_valid_dtypes(tensor_params):
+    """This function checks whether dtypes are supported by the NPU"""
+    supported_dtypes = (np.uint8, np.int8)
+    for tep in tensor_params:
+        # Check for dtypes
+        if np.dtype(tep.dtype) not in supported_dtypes:
+            return False
+        # Check for shape sizes
+        if any(dimlen > 65536 for dimlen in tep.shape):
+            return False
+    return True
+
+
+def check_weights(weights, dilation):
+    """This function checks whether weight tensor is compatible with the NPU"""
+    dilated_height_range = (1, 64)
+    dilated_hxw_range = (1, 64 * 64)
+    weights_limit = 127 * 65536
+    dilated_width = (weights.shape[get_dim_value(weights.layout, "W")] - 1) * dilation[0] + 1
+    dilated_height = (weights.shape[get_dim_value(weights.layout, "H")] - 1) * dilation[1] + 1
+    dh_min, dh_max = dilated_height_range
+    if not dh_min <= dilated_height <= dh_max:
+        return False
+    dilated_hxw = dilated_height * dilated_width
+    dhxw_min, dhxw_max = dilated_hxw_range
+    if not dhxw_min <= dilated_hxw <= dhxw_max:
+        return False
+    # A saturation upper bound check for accumulators
+    weights.values = weights.values - weights.q_params.zero_point
+    axis = (
+        get_dim_value(weights.layout, "H"),
+        get_dim_value(weights.layout, "W"),
+        get_dim_value(weights.layout, "I"),
+    )
+    sum_weights = np.amax(np.sum(np.absolute(weights.values), axis=axis))
+    if not sum_weights <= weights_limit:
+        return False
+    return True
+
+
+def check_bias(bias):
+    """This function checks whether the bias values fit in 40 bits"""
+    if bias and bias.dtype == np.dtype("int64"):
+        valid = all(len(bin(bias_value)[2:]) <= 40 for bias_value in bias.values)
+        return valid
+    return True
+
+
+def check_batch_size(ifm):
+    """This function checks for the number of batches vela currently supports"""
+    if ifm.shape[0] != 1:
+        return False
+    return True
+
+
+def check_dilation(dilation):
+    """This function checks whether dilation is within the limits supported by the NPU"""
+    dilation_range = (1, 2)
+    dmin, dmax = dilation_range
+    if not dmin <= dilation[0] <= dmax:
+        return False
+    if not dmin <= dilation[1] <= dmax:
+        return False
+    return True
+
+
+def check_padding(padding, bounds):
+    """This function checks whether padding is within the limits supported by the NPU"""
+    if len(padding) != 4 or len(bounds) != 4:
+        return False
+    top, left, bottom, right = padding
+    topb, leftb, bottomb, rightb = bounds
+    if top > topb or left > leftb or bottom > bottomb or right > rightb:
+        return False
+    return True
+
+
+class TensorParams:
+    """
+    This class will parse a tvm Expr along with quantization scale
+    and zero point to populate parameters that are required
+    for the creation of tensors in Vela.
+    """
+
+    def __init__(self, tensor, layout=None, scale=None, zero_point=None):
+        self.tensor = tensor
+        if isinstance(tensor, Constant):
+            self.values = tensor.data.asnumpy()
+        else:
+            self.values = None
+        self.dtype = tensor.checked_type.dtype
+        self.shape = [int(i) for i in tensor.checked_type.shape]
+        self.layout = layout
+
+        if scale is not None and zero_point is not None:
+            self.q_params = vapi.NpuQuantization(
+                scale.data.asnumpy().astype("float32"), zero_point.data.asnumpy().astype(self.dtype)
+            )
+        else:
+            # put default values
+            self.q_params = vapi.NpuQuantization(1.0, 0)
+
+
+class QnnConv2DParams:
+    """
+    This class will parse a Call to a ethosu.qnn_conv2d composite function
+    and extract quantization information of all the associated tensors.
+    """
+
+    composite_name = "ethosu.qnn_conv2d"
+    # The NPU only supports padding upto the numbers as follows
+    padding_bounds = [31, 31, 32, 32]
+    activation_map = {"clip": "CLIP"}
+
+    def __init__(self, func_body):
+        activation = None
+        if str(func_body.op) in self.activation_map.keys():
+            activation = func_body
+            requantize_op = activation.args[0]
+        else:
+            requantize_op = func_body
+        bias_add = requantize_op.args[0]
+        qnn_conv2d = bias_add.args[0]
+        data_layout = qnn_conv2d.attrs.data_layout
+        kernel_layout = qnn_conv2d.attrs.kernel_layout
+        # We consider the weights & biases as params as it should be a Constant
+        self.weights = TensorParams(
+            qnn_conv2d.args[QConv2DArgs.WEIGHTS.value],
+            kernel_layout,
+            qnn_conv2d.args[QConv2DArgs.WEIGHTS_SCALE.value],
+            qnn_conv2d.args[QConv2DArgs.WEIGHTS_ZERO_POINT.value],
+        )
+
+        self.biases = TensorParams(
+            bias_add.args[BiasAddArgs.BIASES.value],
+            data_layout,
+            requantize_op.args[RequantArgs.IFM_SCALE.value],
+            requantize_op.args[RequantArgs.IFM_ZERO_POINT.value],
+        )
+        self.ifm = TensorParams(
+            qnn_conv2d.args[QConv2DArgs.IFM.value],
+            data_layout,
+            qnn_conv2d.args[QConv2DArgs.IFM_SCALE.value],
+            qnn_conv2d.args[QConv2DArgs.IFM_ZERO_POINT.value],
+        )
+        self.ofm = TensorParams(
+            func_body,
+            data_layout,
+            requantize_op.args[RequantArgs.OFM_SCALE.value],
+            requantize_op.args[RequantArgs.OFM_ZERO_POINT.value],
+        )
+        self.padding = qnn_conv2d.attrs.padding
+        self.strides = qnn_conv2d.attrs.strides
+        self.dilation = qnn_conv2d.attrs.dilation
+        self.activation = activation
+
+        # If groups are equal to channel, its a depthwise_conv2d
+        self.groups = qnn_conv2d.attrs.groups
+        self.is_depthwise = False
+        channels_axis = {"HWIO": 3, "HWOI": 2}
+        if qnn_conv2d.attrs.groups == self.weights.shape[channels_axis[kernel_layout]]:
+            self.is_depthwise = True
+
+    def is_valid(self):
+        """
+        This function checks whether QnnConv2D has compatible attributes with the NPU
+        """
+        tensor_params = [self.weights, self.ifm, self.ofm]
+        if not check_valid_dtypes(tensor_params):
+            return False
+        if not check_weights(self.weights, self.dilation):
+            return False
+        if not check_bias(self.biases):
+            return False
+        if not check_strides(self.strides):
+            return False
+        if not check_batch_size(self.ifm):
+            return False
+        if not check_dilation(self.dilation):
+            return False
+        if not check_padding(self.padding, self.padding_bounds):
+            return False
+        legal_groups = [1, self.ofm.shape[3]]
+        if self.groups not in legal_groups:
+            return False
+        # This should be a valid QnnDepthwise2DParams, not QnnConv2DParams
+        if self.is_depthwise:
+            return False
+        return True

Review comment:
       ```suggestion
           return not self.is_depthwise
   ```

##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU supported operators."""
+import numpy as np
+
+from tvm.relay.expr import Constant
+from tvm.relay.op.contrib.register import register_pattern_table
+from tvm.relay.dataflow_pattern import wildcard, is_op, is_constant
+from tvm.relay.backend.contrib.ethosu.util import QConv2DArgs
+from tvm.relay.backend.contrib.ethosu.util import BiasAddArgs
+from tvm.relay.backend.contrib.ethosu.util import RequantArgs
+from tvm.relay.backend.contrib.ethosu.util import get_dim_value
+from ethosu.vela import api as vapi
+
+
+def check_strides(strides):
+    """This function checks whether strides are within the limits supported by the NPU"""
+    stride_range = (1, 3)
+    smin, smax = stride_range
+    if not smax >= strides[0] >= smin:
+        return False
+    if not smax >= strides[1] >= smin:
+        return False
+    return True
+
+
+def check_valid_dtypes(tensor_params):
+    """This function checks whether dtypes are supported by the NPU"""
+    supported_dtypes = (np.uint8, np.int8)
+    for tep in tensor_params:
+        # Check for dtypes
+        if np.dtype(tep.dtype) not in supported_dtypes:
+            return False
+        # Check for shape sizes
+        if any(dimlen > 65536 for dimlen in tep.shape):
+            return False
+    return True
+
+
+def check_weights(weights, dilation):
+    """This function checks whether weight tensor is compatible with the NPU"""
+    dilated_height_range = (1, 64)
+    dilated_hxw_range = (1, 64 * 64)
+    weights_limit = 127 * 65536
+    dilated_width = (weights.shape[get_dim_value(weights.layout, "W")] - 1) * dilation[0] + 1
+    dilated_height = (weights.shape[get_dim_value(weights.layout, "H")] - 1) * dilation[1] + 1
+    dh_min, dh_max = dilated_height_range
+    if not dh_min <= dilated_height <= dh_max:
+        return False
+    dilated_hxw = dilated_height * dilated_width
+    dhxw_min, dhxw_max = dilated_hxw_range
+    if not dhxw_min <= dilated_hxw <= dhxw_max:
+        return False
+    # A saturation upper bound check for accumulators
+    weights.values = weights.values - weights.q_params.zero_point
+    axis = (
+        get_dim_value(weights.layout, "H"),
+        get_dim_value(weights.layout, "W"),
+        get_dim_value(weights.layout, "I"),
+    )
+    sum_weights = np.amax(np.sum(np.absolute(weights.values), axis=axis))
+    if not sum_weights <= weights_limit:
+        return False
+    return True
+
+
+def check_bias(bias):
+    """This function checks whether the bias values fit in 40 bits"""
+    if bias and bias.dtype == np.dtype("int64"):
+        valid = all(len(bin(bias_value)[2:]) <= 40 for bias_value in bias.values)
+        return valid
+    return True
+
+
+def check_batch_size(ifm):
+    """This function checks for the number of batches vela currently supports"""
+    if ifm.shape[0] != 1:
+        return False
+    return True
+
+
+def check_dilation(dilation):
+    """This function checks whether dilation is within the limits supported by the NPU"""
+    dilation_range = (1, 2)
+    dmin, dmax = dilation_range
+    if not dmin <= dilation[0] <= dmax:
+        return False
+    if not dmin <= dilation[1] <= dmax:
+        return False
+    return True
+
+
+def check_padding(padding, bounds):
+    """This function checks whether padding is within the limits supported by the NPU"""
+    if len(padding) != 4 or len(bounds) != 4:
+        return False
+    top, left, bottom, right = padding
+    topb, leftb, bottomb, rightb = bounds
+    if top > topb or left > leftb or bottom > bottomb or right > rightb:
+        return False
+    return True

Review comment:
       ```suggestion
       return not (top > topb or left > leftb or bottom > bottomb or right > rightb)
   ```

##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU supported operators."""
+import numpy as np
+
+from tvm.relay.expr import Constant
+from tvm.relay.op.contrib.register import register_pattern_table
+from tvm.relay.dataflow_pattern import wildcard, is_op, is_constant
+from tvm.relay.backend.contrib.ethosu.util import QConv2DArgs
+from tvm.relay.backend.contrib.ethosu.util import BiasAddArgs
+from tvm.relay.backend.contrib.ethosu.util import RequantArgs
+from tvm.relay.backend.contrib.ethosu.util import get_dim_value
+from ethosu.vela import api as vapi
+
+
+def check_strides(strides):
+    """This function checks whether strides are within the limits supported by the NPU"""
+    stride_range = (1, 3)
+    smin, smax = stride_range
+    if not smax >= strides[0] >= smin:
+        return False
+    if not smax >= strides[1] >= smin:
+        return False
+    return True
+
+
+def check_valid_dtypes(tensor_params):
+    """This function checks whether dtypes are supported by the NPU"""
+    supported_dtypes = (np.uint8, np.int8)
+    for tep in tensor_params:
+        # Check for dtypes
+        if np.dtype(tep.dtype) not in supported_dtypes:
+            return False
+        # Check for shape sizes
+        if any(dimlen > 65536 for dimlen in tep.shape):
+            return False
+    return True
+
+
+def check_weights(weights, dilation):
+    """This function checks whether weight tensor is compatible with the NPU"""
+    dilated_height_range = (1, 64)
+    dilated_hxw_range = (1, 64 * 64)
+    weights_limit = 127 * 65536
+    dilated_width = (weights.shape[get_dim_value(weights.layout, "W")] - 1) * dilation[0] + 1
+    dilated_height = (weights.shape[get_dim_value(weights.layout, "H")] - 1) * dilation[1] + 1
+    dh_min, dh_max = dilated_height_range
+    if not dh_min <= dilated_height <= dh_max:
+        return False
+    dilated_hxw = dilated_height * dilated_width
+    dhxw_min, dhxw_max = dilated_hxw_range
+    if not dhxw_min <= dilated_hxw <= dhxw_max:
+        return False
+    # A saturation upper bound check for accumulators
+    weights.values = weights.values - weights.q_params.zero_point
+    axis = (
+        get_dim_value(weights.layout, "H"),
+        get_dim_value(weights.layout, "W"),
+        get_dim_value(weights.layout, "I"),
+    )
+    sum_weights = np.amax(np.sum(np.absolute(weights.values), axis=axis))
+    if not sum_weights <= weights_limit:
+        return False
+    return True
+
+
+def check_bias(bias):
+    """This function checks whether the bias values fit in 40 bits"""
+    if bias and bias.dtype == np.dtype("int64"):
+        valid = all(len(bin(bias_value)[2:]) <= 40 for bias_value in bias.values)
+        return valid
+    return True
+
+
+def check_batch_size(ifm):
+    """This function checks for the number of batches vela currently supports"""
+    if ifm.shape[0] != 1:
+        return False
+    return True

Review comment:
       ```suggestion
       return ifm.shape[0] == 1
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,218 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel, no-value-for-parameter
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm import ir
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """This rewriting converts split operations into a sequence of
+    strided_slice operations, because codegen is going to be based
+    on strided_slices that will define the slice of the tensor that
+    will be fed to the consumer.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split operator takes an array of indices or an integer
+        indicating the number of splits. However, its an array of indices could
+        represent both cases, therefore this function just make it an array of
+        indices where each index represent the co-ordinate of beginning of each
+        section -- defines as section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The Relay Call expression for a split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+@ir.transform.module_pass(opt_level=1)
+class LegalizeSplit:
+    """This is the pass that wraps SplitRewriter"""
+
+    def transform_module(self, mod, ctx):
+        for gv, func in mod.functions.items():
+            func = rewrite(SplitRewriter(), func)
+            mod.update_func(gv, func)
+        return mod
+
+    def __call__(self, *args, **kwargs):

Review comment:
       Do we need this empty call?

##########
File path: tests/python/contrib/test_ethosu/relay_ir_builder.py
##########
@@ -0,0 +1,295 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Helper module to build relay operations for testing"""
+
+from pathlib import Path
+import numpy as np
+import math
+
+import tvm
+from tvm import relay
+from tvm.relay.op.contrib import get_pattern_table
+from tvm.relay import qnn
+from tvm.relay.backend.contrib.ethosu.util import get_range_for_dtype_str
+
+
+class TensorType:

Review comment:
       It'd be great to use `NamedTuple` to clean up this testing code to set the data on instantiation similar to https://github.com/apache/tvm/blob/main/tests/python/relay/aot/aot_test_utils.py#L47

##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU supported operators."""
+import numpy as np
+
+from tvm.relay.expr import Constant
+from tvm.relay.op.contrib.register import register_pattern_table
+from tvm.relay.dataflow_pattern import wildcard, is_op, is_constant
+from tvm.relay.backend.contrib.ethosu.util import QConv2DArgs
+from tvm.relay.backend.contrib.ethosu.util import BiasAddArgs
+from tvm.relay.backend.contrib.ethosu.util import RequantArgs
+from tvm.relay.backend.contrib.ethosu.util import get_dim_value
+from ethosu.vela import api as vapi
+
+
+def check_strides(strides):
+    """This function checks whether strides are within the limits supported by the NPU"""
+    stride_range = (1, 3)
+    smin, smax = stride_range
+    if not smax >= strides[0] >= smin:
+        return False
+    if not smax >= strides[1] >= smin:
+        return False
+    return True
+
+
+def check_valid_dtypes(tensor_params):
+    """This function checks whether dtypes are supported by the NPU"""
+    supported_dtypes = (np.uint8, np.int8)
+    for tep in tensor_params:
+        # Check for dtypes
+        if np.dtype(tep.dtype) not in supported_dtypes:
+            return False
+        # Check for shape sizes
+        if any(dimlen > 65536 for dimlen in tep.shape):
+            return False
+    return True
+
+
+def check_weights(weights, dilation):
+    """This function checks whether weight tensor is compatible with the NPU"""
+    dilated_height_range = (1, 64)
+    dilated_hxw_range = (1, 64 * 64)
+    weights_limit = 127 * 65536
+    dilated_width = (weights.shape[get_dim_value(weights.layout, "W")] - 1) * dilation[0] + 1
+    dilated_height = (weights.shape[get_dim_value(weights.layout, "H")] - 1) * dilation[1] + 1
+    dh_min, dh_max = dilated_height_range
+    if not dh_min <= dilated_height <= dh_max:
+        return False
+    dilated_hxw = dilated_height * dilated_width
+    dhxw_min, dhxw_max = dilated_hxw_range
+    if not dhxw_min <= dilated_hxw <= dhxw_max:
+        return False
+    # A saturation upper bound check for accumulators
+    weights.values = weights.values - weights.q_params.zero_point
+    axis = (
+        get_dim_value(weights.layout, "H"),
+        get_dim_value(weights.layout, "W"),
+        get_dim_value(weights.layout, "I"),
+    )
+    sum_weights = np.amax(np.sum(np.absolute(weights.values), axis=axis))
+    if not sum_weights <= weights_limit:
+        return False
+    return True

Review comment:
       ```suggestion
       return sum_weights <= weights_limit 
   ```

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,218 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel, no-value-for-parameter
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm import ir
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """This rewriting converts split operations into a sequence of
+    strided_slice operations, because codegen is going to be based
+    on strided_slices that will define the slice of the tensor that
+    will be fed to the consumer.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split operator takes an array of indices or an integer
+        indicating the number of splits. However, its an array of indices could
+        represent both cases, therefore this function just make it an array of
+        indices where each index represent the co-ordinate of beginning of each
+        section -- defines as section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The Relay Call expression for a split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+@ir.transform.module_pass(opt_level=1)
+class LegalizeSplit:
+    """This is the pass that wraps SplitRewriter"""
+
+    def transform_module(self, mod, ctx):
+        for gv, func in mod.functions.items():
+            func = rewrite(SplitRewriter(), func)
+            mod.update_func(gv, func)
+        return mod
+
+    def __call__(self, *args, **kwargs):
+        pass
+
+
+class EthosUConv2DRewriter(DFPatternCallback):
+    """Convert conv2d related composite functions into ethosu_conv2d operators"""
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.pattern = (wildcard().has_attr({"Composite": "ethosu.qnn_conv2d"}))(wildcard())
+
+    def callback(self, pre, post, node_map):
+        params = ethosu_patterns.QnnConv2DParams(post.op.body)
+        params.ifm.tensor = post.args[0]
+        channels_map = {
+            "NHWC": 3,
+        }
+        if str(params.ofm.layout) not in channels_map.keys():
+            raise UnsupportedLayout(str(params.ofm.layout))
+        kernel_size_map = {
+            "HWIO": params.weights.shape[0:2],
+            "OHWI": params.weights.shape[1:3],
+            "HWOI": params.weights.shape[0:2],
+        }
+        if str(params.weights.layout) not in kernel_size_map.keys():
+            raise UnsupportedLayout(str(params.weights.layout))
+        activation_map = {"clip": "CLIP"}
+        weight_to_ohwi_transform_map = {"HWIO": [3, 0, 1, 2]}
+        weights_values = params.weights.values
+        weights_values_ohwi = np.transpose(
+            weights_values, weight_to_ohwi_transform_map[str(params.weights.layout)]
+        )
+        if params.activation:
+            activation = activation_map[params.activation.op.name]
+            clip_min = int(params.activation.attrs.a_min)
+            clip_max = int(params.activation.attrs.a_max)
+        else:
+            activation = "NONE"
+            clip_min = 0
+            clip_max = 0
+        scale_bias = vela_api.pack_biases(
+            biases=params.biases.tensor.data.asnumpy(),
+            ifm_scale=params.ifm.q_params.scale_f32,
+            ifm_dtype=np.dtype(params.ifm.dtype),
+            weight_scales=params.weights.q_params.scale_f32,
+            ofm_scale=params.ofm.q_params.scale_f32,
+            is_activation_tanh_or_sigmoid=activation in ["TANH", "SIGMOID"],
+        )
+        ethosu_conv2d = ethosu_ops.ethosu_conv2d(
+            ifm=post.args[0],
+            weight=relay.const(weights_values_ohwi, params.weights.values.dtype),
+            scale_bias=relay.const(scale_bias, "uint8"),
+            lut=relay.const([], dtype="int8"),
+            ifm_scale=float(params.ifm.q_params.scale_f32),
+            ifm_zero_point=int(params.ifm.q_params.zero_point),
+            weight_zero_point=int(params.weights.q_params.zero_point),
+            ofm_scale=float(params.ofm.q_params.scale_f32),
+            ofm_zero_point=int(params.ofm.q_params.zero_point),
+            kernel_shape=kernel_size_map[str(params.weights.layout)],
+            ofm_channels=params.ofm.shape[channels_map[str(params.ofm.layout)]],
+            strides=params.strides,
+            padding=params.padding,
+            dilation=params.dilation,
+            activation=activation,
+            clip_min=clip_min,
+            clip_max=clip_max,
+            upscale="NONE",
+            ifm_layout=str(params.ifm.layout),
+            ofm_layout=str(params.ofm.layout),
+        )
+        return ethosu_conv2d
+
+
+@ir.transform.module_pass(opt_level=1)
+class LegalizeEthosUConv2D:
+    """This is the pass that wraps the EthosUConv2DRewriter"""
+
+    def transform_module(self, mod, ctx):
+        for gv, func in mod.functions.items():

Review comment:
       `gv` again, I think it means `global_var`?

##########
File path: tests/python/contrib/test_ethosu/test_legalize.py
##########
@@ -0,0 +1,333 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument
+
+import pytest
+
+pytest.importorskip("ethosu.vela")
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.backend.contrib import ethosu
+from tvm.relay.backend.contrib.ethosu import legalize, preprocess
+from tvm.relay.dataflow_pattern import *
+from tvm.relay.op.contrib.ethosu import *
+import relay_ir_builder
+
+
+def test_split_indices_legalize():
+    def create_graph(axis):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_relu = relay.nn.relu(x)
+        split_o = relay.split(x_relu, [5, 20, 45], axis).tuple_value
+        return relay.Function([x], split_o)
+
+    def expected_mod_axis1():
+        expected_ir_string = """
+        #[version = "0.0.5"]
+        def @tvmgen_default_ethosu_main_0(%x: Tensor[(1, 50, 50, 3), float32]) -> (Tensor[(1, 5, 50, 3), float32],\
+                                                               Tensor[(1, 15, 50, 3), float32],\
+                                                               Tensor[(1, 25, 50, 3), float32],\
+                                                               Tensor[(1, 5, 50, 3), float32]) {
+          %0 = nn.relu(%x) /* ty=Tensor[(1, 50, 50, 3), float32] */;
+          %1 = strided_slice(%0, begin=[0, 0, 0, 0], end=[1, 5, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 5, 50, 3), float32] */;
+          %2 = strided_slice(%0, begin=[0, 5, 0, 0], end=[1, 20, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 15, 50, 3), float32] */;
+          %3 = strided_slice(%0, begin=[0, 20, 0, 0], end=[1, 45, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 25, 50, 3), float32] */;
+          %4 = strided_slice(%0, begin=[0, 45, 0, 0], end=[1, 50, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 5, 50, 3), float32] */;
+          (%1, %2, %3, %4)
+        }
+        """
+        return tvm.parser.fromtext(expected_ir_string)
+
+    def expected_mod_axis2():
+        expected_ir_string = """
+        #[version = "0.0.5"]
+        def @tvmgen_default_ethosu_main_0(%x: Tensor[(1, 50, 50, 3), float32]) -> (Tensor[(1, 50, 5, 3), float32],\
+                                                               Tensor[(1, 50, 15, 3), float32],\
+                                                               Tensor[(1, 50, 25, 3), float32],\
+                                                               Tensor[(1, 50, 5, 3), float32]) {
+          %0 = nn.relu(%x) /* ty=Tensor[(1, 50, 50, 3), float32] */;
+          %1 = strided_slice(%0, begin=[0, 0, 0, 0], end=[1, 50, 5, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 5, 3), float32] */;
+          %2 = strided_slice(%0, begin=[0, 0, 5, 0], end=[1, 50, 20, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 15, 3), float32] */;
+          %3 = strided_slice(%0, begin=[0, 0, 20, 0], end=[1, 50, 45, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 25, 3), float32] */;
+          %4 = strided_slice(%0, begin=[0, 0, 45, 0], end=[1, 50, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 5, 3), float32] */;
+          (%1, %2, %3, %4)
+        }
+        """
+        return tvm.parser.fromtext(expected_ir_string)
+
+    mod_axis1 = tvm.IRModule()
+    mod_axis1["tvmgen_default_ethosu_main_0"] = create_graph(1)
+    mod_axis1 = legalize.LegalizeSplit()(mod_axis1)
+    expected_axis1 = expected_mod_axis1()
+    tvm.ir.assert_structural_equal(mod_axis1, expected_axis1)
+
+    mod_axis2 = tvm.IRModule()
+    mod_axis2["tvmgen_default_ethosu_main_0"] = create_graph(2)
+    mod_axis2 = legalize.LegalizeSplit()(mod_axis2)
+    expected_axis2 = expected_mod_axis2()
+    tvm.ir.assert_structural_equal(mod_axis2, expected_axis2)
+
+
+def test_split_sections_legalize():
+    def create_graph(axis, sections):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_abs = relay.abs(x)
+        split_o = relay.split(x_abs, sections, axis).tuple_value

Review comment:
       `split_o` ?

##########
File path: src/relay/op/contrib/ethosu/convolution.cc
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/op/contrib/ethosu/convolution.cc
+ * \brief Property def of the Arm(R) Ethos(TM)-U NPU convolution ops.
+ */
+#include "../../nn/convolution.h"
+
+#include <tvm/relay/base.h>
+#include <tvm/relay/op.h>
+#include <tvm/relay/qnn/attrs.h>
+#include <tvm/tir/analysis.h>
+#include <tvm/tir/data_layout.h>
+
+#include "../../../qnn/utils.h"
+#include "common.h"
+
+namespace tvm {
+namespace relay {
+namespace op {
+namespace contrib {
+namespace ethosu {
+
+/*! \brief Attributes used by the Ethos(TM)-U NPU convolution operator */
+struct EthosuConv2DAttrs : public tvm::AttrsNode<EthosuConv2DAttrs> {
+  double ifm_scale;
+  int ifm_zero_point;
+  int weight_zero_point;
+  double ofm_scale;
+  int ofm_zero_point;
+  Array<IndexExpr> kernel_shape;
+  IndexExpr ofm_channels;
+  Array<IndexExpr> strides;
+  Array<IndexExpr> padding;
+  Array<IndexExpr> dilation;
+  String activation;
+  int clip_min;
+  int clip_max;
+  String upscale;
+  tvm::String ifm_layout;
+  tvm::String ofm_layout;
+
+  TVM_DECLARE_ATTRS(EthosuConv2DAttrs, "relay.attrs.EthosuConv2DAttrs") {
+    TVM_ATTR_FIELD(ifm_scale).describe("The quantization scale for the Input Feature Map tensor.");
+    TVM_ATTR_FIELD(ifm_zero_point)
+        .describe("The quantization zero point for the Input Feature Map tensor.");
+    TVM_ATTR_FIELD(weight_zero_point)
+        .describe("The quantization zero point for the weight tensor.");
+    TVM_ATTR_FIELD(ofm_scale).describe("The quantization scale for the Output Feature Map tensor.");
+    TVM_ATTR_FIELD(ofm_zero_point)
+        .describe("The quantization zero point for the Output Feature Map tensor.");
+    TVM_ATTR_FIELD(kernel_shape)
+        .describe("The 2 dimensional kernel shape as (kernel_height, kernel_width).")
+        .set_default(NullValue<Array<IndexExpr> >());

Review comment:
       Weird space here.

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,218 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel, no-value-for-parameter
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm import ir
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """This rewriting converts split operations into a sequence of
+    strided_slice operations, because codegen is going to be based
+    on strided_slices that will define the slice of the tensor that
+    will be fed to the consumer.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split operator takes an array of indices or an integer
+        indicating the number of splits. However, its an array of indices could
+        represent both cases, therefore this function just make it an array of
+        indices where each index represent the co-ordinate of beginning of each
+        section -- defines as section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The Relay Call expression for a split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):

Review comment:
       ```suggestion
           splits_types = {
                   relay.TupleGetItem(post, idx): field_type
                   for idx, field_type in enumerate(post.checked_type.fields)
           }
   ```

##########
File path: src/relay/backend/contrib/ethosu/preprocess.cc
##########
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include <tvm/ir/error.h>
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/attrs/annotation.h>
+#include <tvm/relay/expr.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/transform.h>
+
+#include <unordered_map>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "../../../op/make_op.h"
+
+namespace tvm {
+namespace relay {
+namespace contrib {
+namespace ethosu {
+
+/*!
+ * \brief This expression rewriter will traverse the graph to find calls
+ * to all external functions. If they have multiple inputs and/or
+ * multiple outputs, the following has to be done :
+ * 1) If multiple inputs are present, they needed to be concat before the call.
+ * 2) Inside the external function they need to be split again to their original inputs.
+ * 3) If there are multiple outputs, they need to be concat at the end of external function.
+ * 4) Then, the concat output again need to be split and made the original tuple output in the
+ * main.
+ */
+class ExternalFuncIOHandler : public ExprRewriter {
+ public:
+  explicit ExternalFuncIOHandler(const IRModule& module) : module_(module) {}
+  int count = 0;
+
+  Function InferType(const Function& expr, const IRModule& m) {
+    IRModule mod(m);
+    mod->Update(mod->GetGlobalVar("main"), expr);
+    mod = transform::InferType()(mod);
+    return Downcast<Function>(mod->Lookup("main"));
+  }
+
+  /*!
+   * \brief This function will take shape and compute
+   * the scalar size value for it to be use to create
+   * flat single dimensional tensors.
+   */
+  int64_t CalcSize(const Array<Integer>& shape) {
+    int size = 1;
+    for (auto dim_sz : shape) {

Review comment:
       Can we go for the full `dim_size` here?

##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The relay expression for split operator
+
+        Returns
+        -------
+        section_begins : list

Review comment:
       Should this be `list[int]` ?

##########
File path: tests/python/contrib/test_ethosu/test_legalize.py
##########
@@ -0,0 +1,333 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument
+
+import pytest
+
+pytest.importorskip("ethosu.vela")
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.backend.contrib import ethosu
+from tvm.relay.backend.contrib.ethosu import legalize, preprocess
+from tvm.relay.dataflow_pattern import *
+from tvm.relay.op.contrib.ethosu import *
+import relay_ir_builder
+
+
+def test_split_indices_legalize():
+    def create_graph(axis):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_relu = relay.nn.relu(x)
+        split_o = relay.split(x_relu, [5, 20, 45], axis).tuple_value
+        return relay.Function([x], split_o)
+
+    def expected_mod_axis1():
+        expected_ir_string = """
+        #[version = "0.0.5"]
+        def @tvmgen_default_ethosu_main_0(%x: Tensor[(1, 50, 50, 3), float32]) -> (Tensor[(1, 5, 50, 3), float32],\
+                                                               Tensor[(1, 15, 50, 3), float32],\
+                                                               Tensor[(1, 25, 50, 3), float32],\
+                                                               Tensor[(1, 5, 50, 3), float32]) {
+          %0 = nn.relu(%x) /* ty=Tensor[(1, 50, 50, 3), float32] */;
+          %1 = strided_slice(%0, begin=[0, 0, 0, 0], end=[1, 5, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 5, 50, 3), float32] */;
+          %2 = strided_slice(%0, begin=[0, 5, 0, 0], end=[1, 20, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 15, 50, 3), float32] */;
+          %3 = strided_slice(%0, begin=[0, 20, 0, 0], end=[1, 45, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 25, 50, 3), float32] */;
+          %4 = strided_slice(%0, begin=[0, 45, 0, 0], end=[1, 50, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 5, 50, 3), float32] */;
+          (%1, %2, %3, %4)
+        }
+        """
+        return tvm.parser.fromtext(expected_ir_string)
+
+    def expected_mod_axis2():
+        expected_ir_string = """
+        #[version = "0.0.5"]
+        def @tvmgen_default_ethosu_main_0(%x: Tensor[(1, 50, 50, 3), float32]) -> (Tensor[(1, 50, 5, 3), float32],\
+                                                               Tensor[(1, 50, 15, 3), float32],\
+                                                               Tensor[(1, 50, 25, 3), float32],\
+                                                               Tensor[(1, 50, 5, 3), float32]) {
+          %0 = nn.relu(%x) /* ty=Tensor[(1, 50, 50, 3), float32] */;
+          %1 = strided_slice(%0, begin=[0, 0, 0, 0], end=[1, 50, 5, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 5, 3), float32] */;
+          %2 = strided_slice(%0, begin=[0, 0, 5, 0], end=[1, 50, 20, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 15, 3), float32] */;
+          %3 = strided_slice(%0, begin=[0, 0, 20, 0], end=[1, 50, 45, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 25, 3), float32] */;
+          %4 = strided_slice(%0, begin=[0, 0, 45, 0], end=[1, 50, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 5, 3), float32] */;
+          (%1, %2, %3, %4)
+        }
+        """
+        return tvm.parser.fromtext(expected_ir_string)
+
+    mod_axis1 = tvm.IRModule()
+    mod_axis1["tvmgen_default_ethosu_main_0"] = create_graph(1)
+    mod_axis1 = legalize.LegalizeSplit()(mod_axis1)
+    expected_axis1 = expected_mod_axis1()
+    tvm.ir.assert_structural_equal(mod_axis1, expected_axis1)
+
+    mod_axis2 = tvm.IRModule()
+    mod_axis2["tvmgen_default_ethosu_main_0"] = create_graph(2)
+    mod_axis2 = legalize.LegalizeSplit()(mod_axis2)
+    expected_axis2 = expected_mod_axis2()
+    tvm.ir.assert_structural_equal(mod_axis2, expected_axis2)
+
+
+def test_split_sections_legalize():
+    def create_graph(axis, sections):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_abs = relay.abs(x)
+        split_o = relay.split(x_abs, sections, axis).tuple_value
+        outputs = list()
+        for section_idx in range(sections):
+            split_single_out = relay.TupleGetItem(split_o, section_idx)
+            tanh = relay.tanh(split_single_out)
+            outputs.append(tanh)
+        tuple_out = relay.Tuple(outputs)
+        return relay.Function([x], tuple_out)
+
+    def expected_mod_axis1():
+        expected_ir_string = """
+        #[version = "0.0.5"]
+        def @tvmgen_default_ethosu_main_0(%x: Tensor[(1, 50, 50, 3), float32]) -> (Tensor[(1, 10, 50, 3), float32],\
+                                                               Tensor[(1, 10, 50, 3), float32],\
+                                                               Tensor[(1, 10, 50, 3), float32],\
+                                                               Tensor[(1, 10, 50, 3), float32],\
+                                                               Tensor[(1, 10, 50, 3), float32]) {
+          %0 = abs(%x) /* ty=Tensor[(1, 50, 50, 3), float32] */;
+          %1 = strided_slice(%0, begin=[0, 0, 0, 0], end=[1, 10, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 10, 50, 3), float32] */;
+          %2 = strided_slice(%0, begin=[0, 10, 0, 0], end=[1, 20, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 10, 50, 3), float32] */;
+          %3 = strided_slice(%0, begin=[0, 20, 0, 0], end=[1, 30, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 10, 50, 3), float32] */;
+          %4 = strided_slice(%0, begin=[0, 30, 0, 0], end=[1, 40, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 10, 50, 3), float32] */;
+          %5 = strided_slice(%0, begin=[0, 40, 0, 0], end=[1, 50, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 10, 50, 3), float32] */;
+          %6 = (%1, %2, %3, %4, %5);
+          %7 = %6.0;
+          %8 = tanh(%7) /* ty=Tensor[(1, 10, 50, 3), float32] */;
+          %9 = %6.1;
+          %10 = tanh(%9) /* ty=Tensor[(1, 10, 50, 3), float32] */;
+          %11 = %6.2;
+          %12 = tanh(%11) /* ty=Tensor[(1, 10, 50, 3), float32] */;
+          %13 = %6.3;
+          %14 = tanh(%13) /* ty=Tensor[(1, 10, 50, 3), float32] */;
+          %15 = %6.4;
+          %16 = tanh(%15) /* ty=Tensor[(1, 10, 50, 3), float32] */;
+          (%8, %10, %12, %14, %16)
+        }
+        """
+        return tvm.parser.fromtext(expected_ir_string)
+
+    def expected_mod_axis2():
+        expected_ir_string = """
+        #[version = "0.0.5"]
+        def @tvmgen_default_ethosu_main_0(%x: Tensor[(1, 50, 50, 3), float32]) -> (Tensor[(1, 50, 10, 3), float32],\
+                                                               Tensor[(1, 50, 10, 3), float32],\
+                                                               Tensor[(1, 50, 10, 3), float32],\
+                                                               Tensor[(1, 50, 10, 3), float32],\
+                                                               Tensor[(1, 50, 10, 3), float32]) {
+          %0 = abs(%x) /* ty=Tensor[(1, 50, 50, 3), float32] */;
+          %1 = strided_slice(%0, begin=[0, 0, 0, 0], end=[1, 50, 10, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 10, 3), float32] */;
+          %2 = strided_slice(%0, begin=[0, 0, 10, 0], end=[1, 50, 20, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 10, 3), float32] */;
+          %3 = strided_slice(%0, begin=[0, 0, 20, 0], end=[1, 50, 30, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 10, 3), float32] */;
+          %4 = strided_slice(%0, begin=[0, 0, 30, 0], end=[1, 50, 40, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 10, 3), float32] */;
+          %5 = strided_slice(%0, begin=[0, 0, 40, 0], end=[1, 50, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 10, 3), float32] */;
+          %6 = (%1, %2, %3, %4, %5);
+          %7 = %6.0;
+          %8 = tanh(%7) /* ty=Tensor[(1, 50, 10, 3), float32] */;
+          %9 = %6.1;
+          %10 = tanh(%9) /* ty=Tensor[(1, 50, 10, 3), float32] */;
+          %11 = %6.2;
+          %12 = tanh(%11) /* ty=Tensor[(1, 50, 10, 3), float32] */;
+          %13 = %6.3;
+          %14 = tanh(%13) /* ty=Tensor[(1, 50, 10, 3), float32] */;
+          %15 = %6.4;
+          %16 = tanh(%15) /* ty=Tensor[(1, 50, 10, 3), float32] */;
+          (%8, %10, %12, %14, %16)
+        }
+        """
+        return tvm.parser.fromtext(expected_ir_string)
+
+    mod_axis1 = tvm.IRModule()
+    mod_axis1["tvmgen_default_ethosu_main_0"] = create_graph(1, 5)
+    mod_axis1 = legalize.LegalizeSplit()(mod_axis1)
+    expected_axis1 = expected_mod_axis1()
+    tvm.ir.assert_structural_equal(mod_axis1, expected_axis1)
+
+    mod_axis2 = tvm.IRModule()
+    mod_axis2["tvmgen_default_ethosu_main_0"] = create_graph(2, 5)
+    mod_axis2 = legalize.LegalizeSplit()(mod_axis2)
+    expected_axis2 = expected_mod_axis2()
+    tvm.ir.assert_structural_equal(mod_axis2, expected_axis2)
+
+
+def infer_type_function_pass(func):
+    mod = tvm.IRModule()
+    mod["test"] = func
+    mod = relay.transform.InferType()(mod)
+    return mod["test"]
+
+
+def get_shape_expr(in_expr, out_expr):
+    main_f = relay.Function([in_expr], out_expr)
+    main_f = infer_type_function_pass(main_f)
+    shape = [int(i) for i in main_f.body.checked_type.shape]
+    return shape
+
+
+INVERSE_LAYOUT_TRANSFORM_OHWI_MAP = {
+    "HWIO": [1, 2, 3, 0],
+    "HWOI": [1, 2, 0, 3],
+    "OWHI": [0, 1, 2, 3],
+}
+
+
+def test_ethosu_conv2d_legalize():
+    def create_graph_single(input_tensor_name, input_tensor_shape, input_tensor_dtype):
+        c1_params = relay_ir_builder.QnnConv2DParams(input_tensor_dtype)
+        c1_params.ifm.shape = input_tensor_shape
+        c1_params.kernel.shape = (3, 3, c1_params.ifm.shape[3], 32)
+        c1_params.strides = (1, 1)
+        c1_params.pad = "VALID"
+        c1_params.activation = "CLIP"
+        c1_params.clip_min = 23
+        c1_params.clip_max = 180
+        input0 = relay.var(input_tensor_name, shape=c1_params.ifm.shape, dtype=c1_params.ifm.dtype)
+        c1, new_params = relay_ir_builder.create_qnn_conv2d(c1_params, input0)
+        c1_params.ofm.shape = get_shape_expr(input0, c1)
+
+        f = relay.Function([input0], c1)
+        mod = tvm.IRModule()
+        mod["main"] = f
+        return mod, [c1_params]
+
+    def create_graph_double(input_tensor_name, input_tensor_shape, input_tensor_dtype):
+        c1_params = relay_ir_builder.QnnConv2DParams(input_tensor_dtype)
+        c1_params.ifm.shape = input_tensor_shape
+        c1_params.kernel.shape = (7, 7, c1_params.ifm.shape[3], 8)
+        c1_params.strides = (2, 2)
+        c1_params.pad = "VALID"
+        c1_params.activation = "CLIP"
+        c1_params.clip_min = 10
+        c1_params.clip_max = 240
+        input0 = relay.var(input_tensor_name, shape=c1_params.ifm.shape, dtype=c1_params.ifm.dtype)
+        c1, new_params = relay_ir_builder.create_qnn_conv2d(c1_params, input0)
+        c1_params.ofm.shape = get_shape_expr(input0, c1)
+
+        c2_params = relay_ir_builder.QnnConv2DParams(input_tensor_dtype)
+        c2_params.ifm.shape = c1_params.ofm.shape
+        c2_params.kernel.shape = (5, 5, c2_params.ifm.shape[3], 16)
+        c2_params.strides = (1, 1)
+        c2_params.pad = "SAME"
+        c2, new_params = relay_ir_builder.create_qnn_conv2d(c2_params, c1)
+        c2_params.ofm.shape = get_shape_expr(input0, c2)
+
+        f = relay.Function([input0], c2)
+        mod = tvm.IRModule()
+        mod["main"] = f
+        return mod, [c2_params, c1_params]
+
+    def verify_tensor(tensor_type, expr):
+        assert list(tensor_type.shape) == list(expr.checked_type.shape)
+        assert str(tensor_type.dtype) == str(expr.checked_type.dtype)
+
+    def verify_linear(ext_func, conv2d_params):
+        op = ext_func.body
+        for param in conv2d_params:
+            verify_tensor(param.ifm, op.args[0])
+            verify_tensor(param.ofm, op)
+
+            # This will be in OHWI layout
+            weights_ohwi = op.args[1].data.asnumpy()
+            weights_layout = str(param.kernel.layout)
+            weights = np.transpose(weights_ohwi, INVERSE_LAYOUT_TRANSFORM_OHWI_MAP[weights_layout])
+            assert weights.shape == param.kernel.shape
+            assert weights.dtype == param.kernel.dtype
+
+            assert list(op.args[2].checked_type.shape)[0] == weights_ohwi.shape[0]
+
+            assert float(op.attrs.ifm_scale) == float(param.ifm.sc.data.asnumpy())
+            assert int(op.attrs.ifm_zero_point) == int(param.ifm.zp.data.asnumpy())
+            assert int(op.attrs.weight_zero_point) == int(param.kernel.zp.data.asnumpy())
+            assert float(op.attrs.ofm_scale) == float(param.ofm.sc.data.asnumpy())
+            assert int(op.attrs.ofm_zero_point) == int(param.ofm.zp.data.asnumpy())
+            assert int(op.attrs.ofm_channels) == int(weights_ohwi.shape[0])
+            assert list(op.attrs.padding) == list(param.pad)
+            assert list(op.attrs.strides) == list(param.strides)
+            assert list(op.attrs.dilation) == list(param.dilation)
+            assert str(op.attrs.activation) == str(param.activation)
+            assert int(op.attrs.clip_min) == int(param.clip_min)
+            assert int(op.attrs.clip_max) == int(param.clip_max)
+            op = op.args[0]
+
+    test_cases = [
+        (create_graph_single, ["input", (1, 299, 299, 3), "uint8"]),
+        (create_graph_double, ["input", (1, 128, 256, 4), "uint8"]),
+    ]
+    for test_case in test_cases:
+        mod, conv_params = test_case[0](*test_case[1])
+        mod = ethosu.partition_for_ethosu(mod)
+        mod = legalize.LegalizeEthosUConv2D()(mod)
+        verify_linear(mod["tvmgen_default_ethosu_main_0"], conv_params)
+
+
+def test_ethosu_conv2d_legalize_errors():
+    def create_graph_single_unsupported_ifm_layout(
+        input_tensor_name, input_tensor_shape, input_tensor_dtype
+    ):
+        c1_params = relay_ir_builder.QnnConv2DParams(input_tensor_dtype)
+        c1_params.ifm.shape = input_tensor_shape
+        c1_params.ifm.layout = "NCHW"
+        c1_params.kernel.shape = (3, 3, c1_params.ifm.shape[1], 32)
+        c1_params.strides = (1, 1)
+        c1_params.pad = "VALID"
+        c1_params.activation = "CLIP"
+        c1_params.clip_min = 23
+        c1_params.clip_max = 180
+        input0 = relay.var(input_tensor_name, shape=c1_params.ifm.shape, dtype=c1_params.ifm.dtype)
+        c1, new_params = relay_ir_builder.create_qnn_conv2d(c1_params, input0)
+        c1_params.ofm.shape = get_shape_expr(input0, c1)
+
+        f = relay.Function([input0], c1)
+        mod = tvm.IRModule()
+        mod["main"] = f
+        return mod, [c1_params]
+
+    test_cases = [
+        (create_graph_single_unsupported_ifm_layout, ["input", (1, 3, 299, 299), "uint8"]),
+    ]
+
+    for test_case in test_cases:
+        mod, conv_params = test_case[0](*test_case[1])
+        mod = ethosu.partition_for_ethosu(mod)
+        try:
+            mod = legalize.LegalizeEthosUConv2D()(mod)
+        except Exception as e:
+            assert "EthosUCodegenError: Unsupported Layout NCHW" in e.args[0]

Review comment:
       This is missing the pytest main call:
   ```
   if __name__ == "__main__":
       sys.exit(pytest.main([__file__] + sys.argv[1:]))
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbs-octoml commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-910477523


   re dir structure:  agree given the tight coupling better to keep them together, no need to move around.
   
   I see @Mousius is pushing on the hooks RFC again - I'll try to help get that in place in time for PR4 so we can avoid the bespoke lower te.
   
   Some OctoML folks went through the code (as at PR6) with me yesterday so I understand the te+schedule stuff better now. I would suggest more big-picture comments in both the te and scheduling code since things were not obvious even to experts on our side.
   
   For this PR I think it's just the type annots remaining? That will require a change to mypi.ini to turn it on, but if that unleashes a storm of secondary issues lets reconsider.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r700159189



##########
File path: python/tvm/relay/backend/contrib/ethosu/te/dma.py
##########
@@ -0,0 +1,299 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unnecessary-lambda
+"""Tensor Expressions for operations supported by the DMA engine"""

Review comment:
       I've updated the comment here, but has used NPU in preference to Ethos-U.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r700500587



##########
File path: tests/python/contrib/test_ethosu/relay_ir_builder.py
##########
@@ -0,0 +1,295 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Helper module to build relay operations for testing"""
+
+from pathlib import Path
+import numpy as np
+import math
+
+import tvm
+from tvm import relay
+from tvm.relay.op.contrib import get_pattern_table
+from tvm.relay import qnn
+from tvm.relay.backend.contrib.ethosu.util import get_range_for_dtype_str
+
+
+class TensorType:

Review comment:
       this is one is bit stateful as it uses type inferences and other helpers to populate non-saturating qnn params, post creation of the object.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695434440



##########
File path: python/tvm/relay/backend/contrib/ethosu/__init__.py
##########
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU codegen modules for relay."""

Review comment:
       Done (I think)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r700023373



##########
File path: python/tvm/relay/backend/contrib/ethosu/te/dma.py
##########
@@ -0,0 +1,299 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unnecessary-lambda
+"""Tensor Expressions for operations supported by the DMA engine"""

Review comment:
       Yeah you're pretty much spot on, the TE we create is very closely coupled to the functionality of our NPU and as such we don't want to rely on existing definitions which may be changed. We need it to remain identical so the TIR matching works and we attach some extra information to the TE (usually in the form of attributes) to ease that process.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695436500



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The relay expression for split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+class EthosUConv2DRewriter(DFPatternCallback):
+    """Convert conv2d related composite functions to ethosu_conv2d operators"""
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.pattern = (wildcard().has_attr({"Composite": "ethosu.qnn_conv2d"}))(wildcard())
+
+    def callback(self, pre, post, node_map):
+        params = ethosu_patterns.QnnConv2DParams(post.op.body)
+        params.ifm.tensor = post.args[0]
+        channels_map = {
+            "NHWC": 3,
+        }
+        if str(params.ofm.layout) not in channels_map.keys():
+            raise UnsupportedLayout(str(params.ofm.layout))
+        kernel_size_map = {
+            "HWIO": params.weights.shape[0:2],
+            "OHWI": params.weights.shape[1:3],
+            "HWOI": params.weights.shape[0:2],
+        }
+        if str(params.weights.layout) not in kernel_size_map.keys():
+            raise UnsupportedLayout(str(params.weights.layout))
+        activation_map = {"clip": "CLIP"}
+        weight_to_ohwi_transform_map = {"HWIO": [3, 0, 1, 2]}
+        weights_values = params.weights.values
+        weights_values_ohwi = np.transpose(
+            weights_values, weight_to_ohwi_transform_map[str(params.weights.layout)]
+        )
+        if params.activation:
+            activation = activation_map[params.activation.op.name]
+            clip_min = int(params.activation.attrs.a_min)
+            clip_max = int(params.activation.attrs.a_max)
+        else:
+            activation = "NONE"
+            clip_min = 0
+            clip_max = 0
+        scale_bias = vela_api.pack_biases(
+            biases=params.biases.tensor.data.asnumpy(),
+            ifm_scale=params.ifm.q_params.scale_f32,
+            ifm_dtype=np.dtype(params.ifm.dtype),
+            weight_scales=params.weights.q_params.scale_f32,
+            ofm_scale=params.ofm.q_params.scale_f32,
+            is_activation_tanh_or_sigmoid=activation in ["TANH", "SIGMOID"],
+        )
+        ethosu_conv2d = ethosu_ops.ethosu_conv2d(
+            ifm=post.args[0],
+            weight=relay.const(weights_values_ohwi, params.weights.values.dtype),
+            scale_bias=relay.const(scale_bias, "uint8"),
+            lut=relay.const([], dtype="int8"),
+            ifm_scale=float(params.ifm.q_params.scale_f32),
+            ifm_zero_point=int(params.ifm.q_params.zero_point),
+            weight_zero_point=int(params.weights.q_params.zero_point),
+            ofm_scale=float(params.ofm.q_params.scale_f32),
+            ofm_zero_point=int(params.ofm.q_params.zero_point),
+            kernel_shape=kernel_size_map[str(params.weights.layout)],
+            ofm_channels=params.ofm.shape[channels_map[str(params.ofm.layout)]],
+            strides=params.strides,
+            padding=params.padding,
+            dilation=params.dilation,
+            activation=activation,
+            clip_min=clip_min,
+            clip_max=clip_max,
+            upscale="NONE",
+            ifm_layout=str(params.ifm.layout),
+            ofm_layout=str(params.ofm.layout),
+        )
+        return ethosu_conv2d
+
+
+class LegalizeEthosU:

Review comment:
       Converted to passes.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] NicolaLancellotti commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
NicolaLancellotti commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r696739455



##########
File path: src/relay/op/contrib/ethosu/convolution.cc
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/op/contrib/ethosu/convolution.cc
+ * \brief Property def of the Arm(R) Ethos(TM)-U NPU convolution ops.
+ */
+#include "../../nn/convolution.h"
+
+#include <tvm/relay/base.h>
+#include <tvm/relay/op.h>
+#include <tvm/relay/qnn/attrs.h>
+#include <tvm/tir/analysis.h>
+#include <tvm/tir/data_layout.h>
+
+#include "../../../qnn/utils.h"
+#include "common.h"
+
+namespace tvm {
+namespace relay {
+namespace op {
+namespace contrib {
+namespace ethosu {
+
+/*! \brief Attributes used by the Ethos(TM)-U NPU convolution operator */
+struct EthosuConv2DAttrs : public tvm::AttrsNode<EthosuConv2DAttrs> {
+  double ifm_scale;
+  int ifm_zero_point;
+  int weight_zero_point;
+  double ofm_scale;
+  int ofm_zero_point;
+  Array<IndexExpr> kernel_shape;
+  IndexExpr ofm_channels;
+  Array<IndexExpr> strides;
+  Array<IndexExpr> padding;
+  Array<IndexExpr> dilation;
+  String activation;
+  int clip_min;
+  int clip_max;
+  String upscale;
+  tvm::String ifm_layout;
+  tvm::String ofm_layout;

Review comment:
       ```suggestion
     String ifm_layout;
     String ofm_layout;
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r700174615



##########
File path: python/tvm/relay/backend/contrib/ethosu/te/dma.py
##########
@@ -0,0 +1,299 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unnecessary-lambda
+"""Tensor Expressions for operations supported by the DMA engine"""
+import tvm
+from tvm import te
+from tvm.topi.utils import equal_const_int
+
+
+def _pad_tensor(tensor, pad_before, pad_after=None):
+    """Generate a padded tensor.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to pad.
+    pad_before : tuple of int
+        The 'before' padding on each axis.
+    pad_after : tuple of int
+        The 'after' padding on each axis.
+    Returns
+    -------
+    _pad : callable
+        The padded tensor.
+
+    """
+    pad_after = pad_after or pad_before
+    dims = len(tensor.shape)
+    assert len(pad_before) == dims
+    assert len(pad_after) == dims
+
+    def _pad(*indices):
+        not_zero = []
+        index_tuple = []
+        for i in range(dims):
+            if equal_const_int(pad_before[i], 0) and equal_const_int(pad_after[i], 0):
+                index_tuple.append(indices[i])
+            else:
+                index_tuple.append(indices[i] - pad_before[i])
+                not_zero.append(indices[i] >= pad_before[i])
+                not_zero.append(indices[i] < tensor.shape[i] + pad_before[i])
+        if not_zero:
+            not_zero = tvm.tir.all(*not_zero)
+            return tvm.tir.if_then_else(not_zero, tensor(*index_tuple), tvm.tir.const(0, "uint8"))
+        return tensor(*index_tuple)
+
+    return _pad
+
+
+def read_compute(tensor, layout, zero_point, scale):
+    """A TE compute operator to represent a read.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to read.
+    layout : str
+        The layout of the tensor, either NHWC or NHCWB16.
+    zero_point : int
+        The zero point of the tensor.
+    scale : float
+        The scale of the tensor.
+
+    Returns
+    -------
+    te.Tensor
+        The tensor having been read.
+
+    """
+    assert layout in {"NHWC", "NHCWB16"}
+    read_attrs = {
+        "op": "ethosu_read",
+        "layout": layout,
+        "zero_point": zero_point,
+        "scale": scale,
+    }
+    return te.compute(tensor.shape, lambda *i: tensor(*i), name="ethosu_read", attrs=read_attrs)
+
+
+def write_compute(tensor, layout, zero_point, scale):
+    """A TE compute operator to represent a write.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to write.
+    layout : str
+        The layout of the tensor, either NHWC or NHCWB16.
+    zero_point : int
+        The zero point of the tensor.
+    scale : float
+        The scale of the tensor.
+
+    Returns
+    -------
+    te.Tensor
+        The tensor having been written.
+
+    """
+    assert layout in {"NHWC", "NHCWB16"}
+    write_attrs = {
+        "op": "ethosu_write",
+        "layout": layout,
+        "zero_point": zero_point,
+        "scale": scale,
+    }
+    return te.compute(
+        tensor.shape,
+        lambda *i: tensor(*i),
+        name="ethosu_write",
+        attrs=write_attrs,
+    )
+
+
+def convert_to_nhwc_compute(tensor, layout, channels):
+    """Converts a tensor into NHWC layout if it's in NHWCB16 layout.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to convert.
+    layout : str
+        The layout of the tensor, either NHWC or NHCWB16.
+    channels : int
+        The number of valid channels for the tensor.
+
+    Returns
+    -------
+    te.Tensor
+        The converted tensor in NHWC layout.
+
+    """
+    assert layout in {"NHWC", "NHCWB16"}
+    convert_to_nhwc_attrs = {
+        "op": "ethosu_convert_to_nhwc",
+        "layout": layout,
+    }
+    if layout == "NHCWB16":
+        return te.compute(
+            (tensor.shape[0], tensor.shape[1], tensor.shape[3], channels),
+            lambda nn, hh, ww, cc: tensor(nn, hh, te.indexdiv(cc, 16), ww, te.indexmod(cc, 16)),
+            name="ethosu_convert_to_nhwc",
+            attrs=convert_to_nhwc_attrs,
+        )
+
+    return te.compute(
+        tensor.shape,
+        lambda *i: tensor(*i),
+        name="ethosu_convert_to_nhwc",
+        attrs=convert_to_nhwc_attrs,
+    )
+
+
+def convert_to_nhcwb16_compute(tensor, layout, channels):
+    """Converts a tensor into NHCWB16 layout if it's in NHWC layout.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to convert.
+    layout : str
+        The layout of the tensor, either NHWC or NHCWB16.
+    channels : int
+        The number of valid channels for the tensor.
+
+    Returns
+    -------
+    te.Tensor
+        The converted tensor in NHCWB16 layout.
+
+    """
+    assert layout in {"NHWC", "NHCWB16"}
+    convert_to_nhcwb16_attrs = {
+        "op": "ethosu_convert_to_nhcwb16",
+        "layout": layout,
+    }
+    if layout == "NHCWB16":
+        out_channel_bricks = te.indexdiv(channels - 1, 16) + 1
+        output_shape = (1, tensor.shape[1], out_channel_bricks, tensor.shape[2], 16)
+        return te.compute(
+            output_shape,
+            lambda nn, hh, cc, ww, cb: tvm.tir.if_then_else(
+                cc * 16 + cb < channels,
+                tensor(nn, hh, ww, cc * 16 + cb),
+                tvm.tir.IntImm(tensor.dtype, 0),
+            ),
+            name="ethosu_convert_to_nhcwb16",
+            attrs=convert_to_nhcwb16_attrs,
+        )
+
+    return te.compute(
+        tensor.shape,
+        lambda *i: tensor(*i),
+        name="ethosu_convert_to_nhcwb16",
+        attrs=convert_to_nhcwb16_attrs,
+    )
+
+
+def pad_compute(tensor, padding):
+    """Pad an NHWC tensor in the height and width axes.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to pad.
+    padding : tuple
+        The 4 dimensional padding as (pad_top, pad_left, pad_bottom, pad_right).
+
+    Returns
+    -------
+    te.Tensor
+        The padded tensor.
+
+    """
+    pad_top, pad_left, pad_down, pad_right = padding
+    pad_before = [0, pad_top, pad_left, 0]
+    pad_after = [0, pad_down, pad_right, 0]
+    pad_attrs = {
+        "op": "ethosu_pad",
+    }
+    shape = tensor.shape
+    return te.compute(
+        (shape[0], shape[1] + pad_top + pad_down, shape[2] + pad_left + pad_right, shape[3]),
+        lambda nn, hh, ww, cc: _pad_tensor(tensor, pad_before, pad_after)(nn, hh, ww, cc),
+        name="ethosu_pad",
+        attrs=pad_attrs,
+    )
+
+
+def dma_ifm_compute(ifm, layout, zero_point, scale, channels, padding):
+    """A sequence of compute operators representing the DMA capabilities for an IFM.
+
+    Parameters
+    ----------
+    ifm : te.Tensor
+        The Input Feature Map (IFM) tensor.
+    layout : str
+        The layout of the data, either NHWC or NHCWB16.
+    zero_point : int
+        The zero point of the data.
+    scale : float
+        The scale of the data.
+    channels : int
+        The number of valid channels for the data.
+    padding : Union[int, tuple, list]
+        The desired padding.
+        int -> padding applied to both height and width axes.
+        2D -> padding applied equally on both sides of the (height, width) axes.
+        4D -> padding applied as (top, left, bottom, right)

Review comment:
       Right you are, this comment is out-of-date. I'll fix it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-909087831


   @jroesch @mbs-octoml ,
   
   Sorry have been busy getting all the PRs up and green (which they are now :) . Yes, we were off over long-weekend here in the UK.
   
   Given its python dependency of Vela, we find keeping the integration and passes in python for the NPU lowering is beneficial, especially in debugging. However, we have moved that involves relay traversal implementations (bar the pattern rewrites -- because the python hook is just an interface and I believe the rewriting is actually happening in C++) to C++ while maintaining a python interface to cater the integration. Performance was the consideration to move the pass impl to C++. If we find some passes are slow, we'll put effort to move the impl to C++.
   
   @mbs-octoml 
   
   For 1.)
   
   > Should the op defs go in python/tvm/relay/op/contrib/ethosu.py (where you currently have just the MergeComposite pattern defs), and the te defs should go in python/tvm/topi/ethosu/conv2d.py? We've gotten this far without a contrib under python/tvm/relay/backend so trying to follow existing convention as much as possible even with the custom scheduling would be nice.
   
   We do have a src/relay/backend/contrib/ -- this is just mirroring that folder in python, therefore I think it is less confusing to create it there.
   for op definitions, @mbaret WDYT ? -- Having everything closer to the Ethos-U codegen seems easier to navigate for me.
   
   For 2.)
   
   Yes, we agree to add the type annotations for relay types.
   
   For 3.)
   
   @mbaret , we need that by P3 right?
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] ekalda commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
ekalda commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r694228019



##########
File path: python/tvm/relay/backend/contrib/ethosu/te/convolution.py
##########
@@ -0,0 +1,199 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unused-argument
+"""Tensor Expressions for convolutions for the NPU"""
+from tvm import te
+from .dma import dma_ofm_compute, dma_ifm_compute
+
+
+def process_stride(stride):
+    """Process the striding into a common format.
+
+    Parameters
+    ----------
+    stride : Union[int, tuple, list]
+        The 2D striding.
+        int -> striding is the same in the height and width axis.
+        2D -> striding specified as (stride height, stride width).
+
+    Returns
+    -------
+    int
+        The stride in the height axis.
+    int
+        The stride in the width axis.
+
+    """
+    assert isinstance(stride, int) or len(stride) == 2
+    if isinstance(stride, int):
+        return stride, stride
+
+    return stride
+
+
+def process_dilation(dilation):
+    """Process the dilation into a common format.
+
+    Parameters
+    ----------
+    dilation : Union[int, tuple, list]
+        The 2D dilation.
+        int -> dilation is the same in the height and width axis.
+        2D -> dilation specified as (dilation height, dilation width).
+
+    Returns
+    -------
+    int
+        The dilation in the height axis.
+    int
+        The dilation in the width axis.
+
+    """
+    assert isinstance(dilation, int) or len(dilation) == 2
+    if isinstance(dilation, int):
+        return dilation, dilation
+
+    return dilation

Review comment:
       Looks like these don't get used anywhere? Or if they do, maybe they could be squashed into one function?

##########
File path: python/tvm/relay/backend/contrib/ethosu/te/convolution.py
##########
@@ -0,0 +1,199 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unused-argument
+"""Tensor Expressions for convolutions for the NPU"""
+from tvm import te
+from .dma import dma_ofm_compute, dma_ifm_compute
+
+
+def process_stride(stride):
+    """Process the striding into a common format.
+
+    Parameters
+    ----------
+    stride : Union[int, tuple, list]
+        The 2D striding.
+        int -> striding is the same in the height and width axis.
+        2D -> striding specified as (stride height, stride width).
+
+    Returns
+    -------
+    int
+        The stride in the height axis.
+    int
+        The stride in the width axis.
+
+    """
+    assert isinstance(stride, int) or len(stride) == 2
+    if isinstance(stride, int):
+        return stride, stride
+
+    return stride
+
+
+def process_dilation(dilation):
+    """Process the dilation into a common format.
+
+    Parameters
+    ----------
+    dilation : Union[int, tuple, list]
+        The 2D dilation.
+        int -> dilation is the same in the height and width axis.
+        2D -> dilation specified as (dilation height, dilation width).
+
+    Returns
+    -------
+    int
+        The dilation in the height axis.
+    int
+        The dilation in the width axis.
+
+    """
+    assert isinstance(dilation, int) or len(dilation) == 2
+    if isinstance(dilation, int):
+        return dilation, dilation
+
+    return dilation
+
+
+def conv2d_compute(
+    ifm,
+    weight,
+    scale_bias,
+    lut,
+    ifm_scale,
+    ifm_zero_point,
+    weight_zero_point,
+    ofm_scale,
+    ofm_zero_point,
+    strides,
+    padding,
+    dilation,
+    activation,
+    clip_min,
+    clip_max,
+    upscale,
+    ifm_layout,
+    ofm_layout,
+):
+    """A compute operator representing the capabilities of a 2D convolution for the NPU.
+
+    Parameters
+    ----------
+    ifm : te.Tensor
+        The Input Feature Map tensor (IFM).
+    weight : te.Tensor
+        The weight tensor.
+    scale_bias : te.Tensor
+        The packed per-channel weight scale and bias tensor.
+    lut : te.Tensor
+        The look-up table values to use if activation = "LUT".
+    ifm_scale : float
+        The quantization scale for the Input Feature Map tensor.
+    ifm_zero_point : int
+        The quantization zero point for the Input Feature Map tensor.
+    weight_zero_point : int
+        The quantization zero point for the weight tensor.
+    ofm_scale : float
+        The quantization scale for the Output Feature Map tensor.
+    ofm_zero_point : int
+        The quantization zero point for the Output Feature Map tensor.
+    strides : tuple
+        The 2 dimensional strides as (stride_height, stride_width).
+    padding : tuple
+        The 4 dimensional padding as (pad_top, pad_left, pad_bottom, pad_right).
+    dilation : Union[int, tuple, list]
+        The 2 dimensional dilation as (dilation_height, dilation_width).
+    activation : str
+        The activation function to use.
+            "NONE" - no activation function.
+            "CLIP" - clip the output between clip_min and clip_max.
+            "TANH" - tanh activation function.
+            "SIGMOID" - sigmoid activation function.
+            "LUT" - use a look-up table to perform the activation function.
+    clip_min : int
+        The minimum clipping value if activation = "CLIP".
+    clip_max : int
+        The maximum clipping value if activation = "CLIP".
+    upscale : str
+        The 2x2 upscaling mode to apply to the Input Feature Map tensor.
+            "NONE" - no upscaling.
+            "NEAREST" - upscale using nearest neighbour.
+            "ZEROS" - upscale using zeros.
+    ifm_layout : str
+        The layout of the Input Feature Map tensor. Can be "NHWC" or "NHCWB16".
+    ofm_layout : str
+        The layout of the Output Feature Map tensor. Can be "NHWC" or "NHCWB16".
+
+    Returns
+    -------
+    te.Tensor
+        The OFM tensor.
+
+    """
+    assert ifm.shape[0] == 1
+    assert ifm_layout in {"NHWC", "NHCWB16"}
+    assert ofm_layout in {"NHWC", "NHCWB16"}
+
+    stride_h, stride_w = strides
+    dilation_h, dilation_w = dilation
+    ofm_channels, kernel_h, kernel_w, ifm_channels = weight.shape
+
+    # Compute operation for the IFM DMA pipeline
+    dmaed_ifm = dma_ifm_compute(
+        ifm, ifm_layout, ifm_zero_point, ifm_scale, weight.shape[3], padding
+    )
+
+    # 2D Convolution compute operation
+    dilated_kernel_h = (kernel_h - 1) * dilation_h + 1
+    dilated_kernel_w = (kernel_w - 1) * dilation_w + 1

Review comment:
       It seems to me that finding an output shape based on the kernel size, strides and dilation is a very common operation, so maybe worth creating a separate function?

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn conv2d arguments
+    """
+
+    ifm = 0
+    weights = 1
+    ifm_zero_point = 2
+    weights_zero_point = 3
+    ifm_scale = 4
+    weights_scale = 5
+
+
+class RequantArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn requantize arguments
+    """
+
+    ifm_scale = 1
+    ifm_zero_point = 2
+    ofm_scale = 3
+    ofm_zero_point = 4
+
+
+class BiasAddArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    biases = 1
+
+
+class ClipArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    a_min = 1
+    a_max = 2
+
+
+class MaxPoolArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm = 0
+
+
+class AddArgs(Enum):
+    """This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm0 = 0
+    ifm1 = 1
+    ifm0_scale = 2
+    ifm0_zero_point = 3
+    ifm1_scale = 4
+    ifm1_zero_point = 5
+    ofm_scale = 6
+    ofm_zero_point = 7

Review comment:
       Looks like this class doesn't get used in this commit

##########
File path: python/tvm/relay/backend/contrib/ethosu/te/dma.py
##########
@@ -0,0 +1,299 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unnecessary-lambda
+"""Tensor Expressions for operations supported by the DMA engine"""
+import tvm
+from tvm import te
+from tvm.topi.utils import equal_const_int
+
+
+def _pad_tensor(tensor, pad_before, pad_after=None):
+    """Generate a padded tensor.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to pad.
+    pad_before : tuple of int
+        The 'before' padding on each axis.
+    pad_after : tuple of int
+        The 'after' padding on each axis.
+    Returns
+    -------
+    _pad : callable
+        The padded tensor.
+
+    """
+    pad_after = pad_after or pad_before
+    dims = len(tensor.shape)
+    assert len(pad_before) == dims
+    assert len(pad_after) == dims
+
+    def _pad(*indices):
+        not_zero = []
+        index_tuple = []

Review comment:
       I think it is worth documenting what these variables mean

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn conv2d arguments
+    """
+
+    ifm = 0
+    weights = 1
+    ifm_zero_point = 2
+    weights_zero_point = 3
+    ifm_scale = 4
+    weights_scale = 5
+
+
+class RequantArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn requantize arguments
+    """
+
+    ifm_scale = 1
+    ifm_zero_point = 2
+    ofm_scale = 3
+    ofm_zero_point = 4
+
+
+class BiasAddArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    biases = 1
+
+
+class ClipArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments

Review comment:
       clip

##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU supported operators."""
+import numpy as np
+
+from tvm.relay.expr import Constant
+from tvm.relay.op.contrib.register import register_pattern_table
+from tvm.relay.dataflow_pattern import wildcard, is_op, is_constant
+from tvm.relay.backend.contrib.ethosu.util import QConv2DArgs
+from tvm.relay.backend.contrib.ethosu.util import BiasAddArgs
+from tvm.relay.backend.contrib.ethosu.util import RequantArgs
+from tvm.relay.backend.contrib.ethosu.util import get_dim_value
+from ethosu.vela import api as vapi
+
+
+def check_strides(strides):
+    """Checks whether strides are within the limits supported by the hardware"""
+    stride_range = (1, 3)
+    smin, smax = stride_range
+    if not smax >= strides[0] >= smin:
+        return False
+    if not smax >= strides[1] >= smin:
+        return False
+    return True
+
+
+def check_valid_dtypes(tensor_params):
+    """Check whether dtypes are supported by the hardware"""
+    supported_dtypes = (np.uint8, np.int8)
+    for tep in tensor_params:
+        # Check for dtypes
+        if np.dtype(tep.dtype) not in supported_dtypes:
+            return False
+        # Check for shape sizes
+        if any(dimlen > 65536 for dimlen in tep.shape):
+            return False
+    return True
+
+
+def check_weights(weights, dilation):
+    """Checks whether weight tensor is compatible with HW"""
+    dilated_height_range = (1, 64)
+    dilated_hxw_range = (1, 64 * 64)
+    weights_limit = 127 * 65536
+    dilated_width = (weights.shape[get_dim_value(weights.layout, "W")] - 1) * dilation[0] + 1
+    dilated_height = (weights.shape[get_dim_value(weights.layout, "H")] - 1) * dilation[1] + 1
+    dh_min, dh_max = dilated_height_range
+    if not dh_min <= dilated_height <= dh_max:
+        return False
+    dilated_hxw = dilated_height * dilated_width
+    dhxw_min, dhxw_max = dilated_hxw_range
+    if not dhxw_min <= dilated_hxw <= dhxw_max:
+        return False
+    # A saturation upper bound check for accumulators
+    weights.values = weights.values - weights.q_params.zero_point
+    axis = (
+        get_dim_value(weights.layout, "H"),
+        get_dim_value(weights.layout, "W"),
+        get_dim_value(weights.layout, "I"),
+    )
+    sum_weights = np.amax(np.sum(np.absolute(weights.values), axis=axis))
+    if not sum_weights <= weights_limit:
+        return False
+    return True
+
+
+def check_bias(bias):
+    """Check whether the bias values fit in 40 bits"""
+    if bias and bias.dtype == np.dtype("int64"):
+        valid = all(len(bin(bias_value)[2:]) <= 40 for bias_value in bias.values)
+        return valid
+    return True
+
+
+def check_batch_size(ifm):
+    """Checks for the number of batches vela currently supports"""
+    if ifm.shape[0] != 1:
+        return False
+    return True
+
+
+def check_dilation(dilation):
+    """Checks whether dilation is within the limits supported by the hardware"""
+    dilation_range = (1, 2)
+    dmin, dmax = dilation_range
+    if not dmin <= dilation[0] <= dmax:
+        return False
+    if not dmin <= dilation[1] <= dmax:
+        return False
+    return True
+
+
+def check_padding(padding, bounds):
+    """Checks whether padding is within the limits supported by the hardware"""
+    if len(padding) != 4 or len(bounds) != 4:
+        return False
+    top, left, bottom, right = padding
+    topb, leftb, bottomb, rightb = bounds
+    if top > topb or left > leftb or bottom > bottomb or right > rightb:
+        return False
+    return True
+
+
+class TensorParams:
+    """
+    This class will parse a tvm Expr along with quantization scale
+    and zero point to populate parameters that are required
+    for the creation of tensors in Vela.
+    """
+
+    def __init__(self, tensor, layout=None, scale=None, zero_point=None):
+        self.tensor = tensor
+        if isinstance(tensor, Constant):
+            self.values = tensor.data.asnumpy()
+        else:
+            self.values = None
+        self.dtype = tensor.checked_type.dtype
+        self.shape = [int(i) for i in tensor.checked_type.shape]
+        self.layout = layout
+
+        if scale is not None and zero_point is not None:
+            self.q_params = vapi.NpuQuantization(
+                scale.data.asnumpy().astype("float32"), zero_point.data.asnumpy().astype(self.dtype)
+            )
+        else:
+            # put default values
+            self.q_params = vapi.NpuQuantization(1.0, 0)
+
+
+class QnnConv2DParams:
+    """
+    This class will parse a Call to a ethosu.qnn_conv2d_clip composite function

Review comment:
       Shouldn't this be just ethosu.qnn_conv2d?

##########
File path: python/tvm/relay/backend/contrib/ethosu/te/dma.py
##########
@@ -0,0 +1,299 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unnecessary-lambda
+"""Tensor Expressions for operations supported by the DMA engine"""
+import tvm
+from tvm import te
+from tvm.topi.utils import equal_const_int
+
+
+def _pad_tensor(tensor, pad_before, pad_after=None):
+    """Generate a padded tensor.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to pad.
+    pad_before : tuple of int
+        The 'before' padding on each axis.
+    pad_after : tuple of int
+        The 'after' padding on each axis.
+    Returns
+    -------
+    _pad : callable
+        The padded tensor.
+
+    """
+    pad_after = pad_after or pad_before
+    dims = len(tensor.shape)
+    assert len(pad_before) == dims
+    assert len(pad_after) == dims
+
+    def _pad(*indices):
+        not_zero = []
+        index_tuple = []
+        for i in range(dims):
+            if equal_const_int(pad_before[i], 0) and equal_const_int(pad_after[i], 0):
+                index_tuple.append(indices[i])
+            else:
+                index_tuple.append(indices[i] - pad_before[i])
+                not_zero.append(indices[i] >= pad_before[i])
+                not_zero.append(indices[i] < tensor.shape[i] + pad_before[i])
+        if not_zero:
+            not_zero = tvm.tir.all(*not_zero)
+            return tvm.tir.if_then_else(not_zero, tensor(*index_tuple), tvm.tir.const(0, "uint8"))
+        return tensor(*index_tuple)
+
+    return _pad
+
+
+def read_compute(tensor, layout, zero_point, scale):
+    """A TE compute operator to represent a read.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to read.
+    layout : str
+        The layout of the tensor, either NHWC or NHCWB16.
+    zero_point : int
+        The zero point of the tensor.
+    scale : float
+        The scale of the tensor.
+
+    Returns
+    -------
+    te.Tensor
+        The tensor having been read.
+
+    """
+    assert layout in {"NHWC", "NHCWB16"}
+    read_attrs = {
+        "op": "ethosu_read",
+        "layout": layout,
+        "zero_point": zero_point,
+        "scale": scale,
+    }
+    return te.compute(tensor.shape, lambda *i: tensor(*i), name="ethosu_read", attrs=read_attrs)
+
+
+def write_compute(tensor, layout, zero_point, scale):
+    """A TE compute operator to represent a write.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to write.
+    layout : str
+        The layout of the tensor, either NHWC or NHCWB16.
+    zero_point : int
+        The zero point of the tensor.
+    scale : float
+        The scale of the tensor.
+
+    Returns
+    -------
+    te.Tensor
+        The tensor having been written.
+
+    """
+    assert layout in {"NHWC", "NHCWB16"}
+    write_attrs = {
+        "op": "ethosu_write",
+        "layout": layout,
+        "zero_point": zero_point,
+        "scale": scale,
+    }
+    return te.compute(
+        tensor.shape,
+        lambda *i: tensor(*i),
+        name="ethosu_write",
+        attrs=write_attrs,
+    )
+
+
+def convert_to_nhwc_compute(tensor, layout, channels):
+    """Converts a tensor into NHWC layout if it's in NHWCB16 layout.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to convert.
+    layout : str
+        The layout of the tensor, either NHWC or NHCWB16.
+    channels : int
+        The number of valid channels for the tensor.
+
+    Returns
+    -------
+    te.Tensor
+        The converted tensor in NHWC layout.
+
+    """
+    assert layout in {"NHWC", "NHCWB16"}
+    convert_to_nhwc_attrs = {
+        "op": "ethosu_convert_to_nhwc",
+        "layout": layout,
+    }
+    if layout == "NHCWB16":
+        return te.compute(
+            (tensor.shape[0], tensor.shape[1], tensor.shape[3], channels),
+            lambda nn, hh, ww, cc: tensor(nn, hh, te.indexdiv(cc, 16), ww, te.indexmod(cc, 16)),
+            name="ethosu_convert_to_nhwc",
+            attrs=convert_to_nhwc_attrs,
+        )
+
+    return te.compute(
+        tensor.shape,
+        lambda *i: tensor(*i),
+        name="ethosu_convert_to_nhwc",
+        attrs=convert_to_nhwc_attrs,
+    )
+
+
+def convert_to_nhcwb16_compute(tensor, layout, channels):
+    """Converts a tensor into NHCWB16 layout if it's in NHWC layout.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to convert.
+    layout : str
+        The layout of the tensor, either NHWC or NHCWB16.
+    channels : int
+        The number of valid channels for the tensor.
+
+    Returns
+    -------
+    te.Tensor
+        The converted tensor in NHCWB16 layout.
+
+    """
+    assert layout in {"NHWC", "NHCWB16"}
+    convert_to_nhcwb16_attrs = {
+        "op": "ethosu_convert_to_nhcwb16",
+        "layout": layout,
+    }
+    if layout == "NHCWB16":
+        out_channel_bricks = te.indexdiv(channels - 1, 16) + 1
+        output_shape = (1, tensor.shape[1], out_channel_bricks, tensor.shape[2], 16)
+        return te.compute(
+            output_shape,
+            lambda nn, hh, cc, ww, cb: tvm.tir.if_then_else(
+                cc * 16 + cb < channels,
+                tensor(nn, hh, ww, cc * 16 + cb),
+                tvm.tir.IntImm(tensor.dtype, 0),
+            ),
+            name="ethosu_convert_to_nhcwb16",
+            attrs=convert_to_nhcwb16_attrs,
+        )
+
+    return te.compute(
+        tensor.shape,
+        lambda *i: tensor(*i),
+        name="ethosu_convert_to_nhcwb16",
+        attrs=convert_to_nhcwb16_attrs,
+    )
+
+
+def pad_compute(tensor, padding):
+    """Pad an NHWC tensor in the height and width axes.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to pad.
+    padding : tuple
+        The 4 dimensional padding as (pad_top, pad_left, pad_bottom, pad_right).
+
+    Returns
+    -------
+    te.Tensor
+        The padded tensor.
+
+    """
+    pad_top, pad_left, pad_down, pad_right = padding
+    pad_before = [0, pad_top, pad_left, 0]
+    pad_after = [0, pad_down, pad_right, 0]
+    pad_attrs = {
+        "op": "ethosu_pad",
+    }
+    shape = tensor.shape
+    return te.compute(
+        (shape[0], shape[1] + pad_top + pad_down, shape[2] + pad_left + pad_right, shape[3]),
+        lambda nn, hh, ww, cc: _pad_tensor(tensor, pad_before, pad_after)(nn, hh, ww, cc),
+        name="ethosu_pad",
+        attrs=pad_attrs,
+    )
+
+
+def dma_ifm_compute(ifm, layout, zero_point, scale, channels, padding):
+    """A sequence of compute operators representing the DMA capabilities for an IFM.
+
+    Parameters
+    ----------
+    ifm : te.Tensor
+        The Input Feature Map (IFM) tensor.
+    layout : str
+        The layout of the data, either NHWC or NHCWB16.
+    zero_point : int
+        The zero point of the data.
+    scale : float
+        The scale of the data.
+    channels : int
+        The number of valid channels for the data.
+    padding : Union[int, tuple, list]
+        The desired padding.
+        int -> padding applied to both height and width axes.
+        2D -> padding applied equally on both sides of the (height, width) axes.
+        4D -> padding applied as (top, left, bottom, right)

Review comment:
       Looks like pad_compute only handles 4D padding though




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-910615699


   I've addressed/anwsered existing comments, we ll take pass tommorow add type annotations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-909087831


   @jroesch @mbs-octoml ,
   
   Sorry have been busy getting all the PRs up and green (which they are now :) . Yes, we were off over long-weekend here in the UK.
   
   Given its python dependency of Vela, we find keeping the integration and passes in python for the NPU lowering is beneficial, especially in debugging. However, we have moved that involves relay traversal implementations (bar the pattern rewrites -- because the python hook is just an interface and I believe the rewriting is actually happening in C++) to C++ while maintaining a python interface to cater the integration. Performance was the consideration to move the pass impl to C++. If we find some passes are slow, we'll put effort to move the impl to C++.
   
   @mbs-octoml 
   
   For 1.)
   
   > Should the op defs go in python/tvm/relay/op/contrib/ethosu.py (where you currently have just the MergeComposite pattern defs), and the te defs should go in python/tvm/topi/ethosu/conv2d.py? We've gotten this far without a contrib under python/tvm/relay/backend so trying to follow existing convention as much as possible even with the custom scheduling would be nice.
   
   We do have a src/relay/backend/contrib/ -- this is just mirroring that folder in python, therefore I think it is less confusing to create it there.
   for op definitions, @mbaret WDYT ? -- Having everything closer to the Ethos-U codegen seems easier to navigate for me.
   
   For 2.)
   
   Yes, we agree to add the type annotations for relay types.
   
   For 3.)
   
   @mbaret , we need that by P3 right?
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r700500936



##########
File path: tests/python/contrib/test_ethosu/test_legalize.py
##########
@@ -0,0 +1,333 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument
+
+import pytest
+
+pytest.importorskip("ethosu.vela")
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.backend.contrib import ethosu
+from tvm.relay.backend.contrib.ethosu import legalize, preprocess
+from tvm.relay.dataflow_pattern import *
+from tvm.relay.op.contrib.ethosu import *
+import relay_ir_builder
+
+
+def test_split_indices_legalize():
+    def create_graph(axis):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_relu = relay.nn.relu(x)
+        split_o = relay.split(x_relu, [5, 20, 45], axis).tuple_value

Review comment:
       just variable name to say output of split ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r700501642



##########
File path: src/relay/op/contrib/ethosu/convolution.cc
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/op/contrib/ethosu/convolution.cc
+ * \brief Property def of the Arm(R) Ethos(TM)-U NPU convolution ops.
+ */
+#include "../../nn/convolution.h"
+
+#include <tvm/relay/base.h>
+#include <tvm/relay/op.h>
+#include <tvm/relay/qnn/attrs.h>
+#include <tvm/tir/analysis.h>
+#include <tvm/tir/data_layout.h>
+
+#include "../../../qnn/utils.h"
+#include "common.h"
+
+namespace tvm {
+namespace relay {
+namespace op {
+namespace contrib {
+namespace ethosu {
+
+/*! \brief Attributes used by the Ethos(TM)-U NPU convolution operator */
+struct EthosuConv2DAttrs : public tvm::AttrsNode<EthosuConv2DAttrs> {
+  double ifm_scale;
+  int ifm_zero_point;
+  int weight_zero_point;
+  double ofm_scale;
+  int ofm_zero_point;
+  Array<IndexExpr> kernel_shape;
+  IndexExpr ofm_channels;
+  Array<IndexExpr> strides;
+  Array<IndexExpr> padding;
+  Array<IndexExpr> dilation;
+  String activation;
+  int clip_min;
+  int clip_max;
+  String upscale;
+  tvm::String ifm_layout;
+  tvm::String ofm_layout;

Review comment:
       @mbaret, can you take a look at this one?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-910118010


   Regarding point 3, it is indeed required for P3. Having the relay_to_tir extension alone wouldn't be sufficient to get rid of the custom lower_to_te pass, we'd need a relay->TE (unscheduled) lowerer specifically. If that does happen to land in time though, I'd be happy to upgrade to it. Otherwise, I'm also OK with migrating as soon as it becomes available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695438174



##########
File path: python/tvm/relay/backend/contrib/ethosu/vela_api.py
##########
@@ -0,0 +1,314 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+conversions between TVM and Vela. Therefore, all interactions with the
+Vela API are supposed to go through this adapter, with the hope that
+any changes to Vela API, TVM only needs to change this file.
+The following conversion APIs are added :
+    *Obtaining the best block config
+    *Compressing weights
+    *Packing biases
+"""
+import logging
+import math
+import numpy as np
+from ethosu.vela import api as vapi
+
+from tvm.relay.backend.contrib.ethosu import util
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("Ethos-U")
+
+VELA_TO_NP_DTYPES = {
+    vapi.NpuDataType.UINT8: np.uint8,
+    vapi.NpuDataType.UINT16: np.uint16,
+    vapi.NpuDataType.INT8: np.int8,
+    vapi.NpuDataType.INT16: np.int16,
+    vapi.NpuDataType.INT32: np.int32,
+}
+
+SCALE_BIAS_LENGTH = 10
+
+
+def get_optimal_block_config(npu_op, accel_type):
+    """
+    "The NPU's unit of work is known as a block. It will fetch block(s) from Input
+    Feature Map (IFM) and a compute block for Output Feature Map (OFM).
+    Therefore, we need to pick an optimal block configuration considering bandwidth
+    to bring IFM blocks and the number of OFM block computes need to happen
+    to cover the OFM as indicated by the npu op.
+
+    Parameters
+    ----------
+    npu_op : ethosu.vela.api.NpuOperation
+        The NPU operation and its params
+    accel_type : ethosu.vela.api.NpuAccelerator
+        The NPU accelerator variant
+    Returns
+    -------
+    ethosu.vela.api.NpuShape3d :
+        The optimal block config for the operator
+    """
+    all_valid_block_configs = vapi.npu_find_block_configs(npu_op, accel_type)
+    return _get_optimal_block_config(all_valid_block_configs)
+
+
+def _get_optimal_block_config(all_valid_block_configs):
+    """An internal function to get block config with largest depth
+    and then highest volume/area"""
+    assert isinstance(all_valid_block_configs, list)
+    for block_cfg in all_valid_block_configs:
+        assert isinstance(block_cfg, vapi.NpuShape3D)
+
+    # Getting the largest volume block for benchmarksing
+    all_valid_block_configs.sort(
+        key=lambda _cfg: _cfg.depth * _cfg.height * _cfg.width, reverse=True
+    )
+    largest_volume_block_config = all_valid_block_configs[0]
+    largest_volume = (
+        largest_volume_block_config.depth
+        * largest_volume_block_config.height
+        * largest_volume_block_config.width
+    )
+
+    all_valid_block_configs.sort(key=lambda _cfg: _cfg.depth, reverse=True)
+    max_d = all_valid_block_configs[0].depth
+    max_depth_block_configs = [_cfg for _cfg in all_valid_block_configs if _cfg.depth == max_d]
+    max_depth_block_configs.sort(key=lambda _cfg: _cfg.height * _cfg.width, reverse=True)
+    max_area = max_depth_block_configs[0].height * max_depth_block_configs[0].width
+    max_area_depth_block_configs = [
+        _cfg for _cfg in max_depth_block_configs if _cfg.height * _cfg.width == max_area
+    ]
+    # This to get a deterministic anwser everytime
+    max_area_depth_block_configs.sort(key=lambda _cfg: _cfg.height, reverse=True)
+    assert len(max_area_depth_block_configs) > 0
+    current_volume = (
+        max_area_depth_block_configs[0].depth
+        * max_area_depth_block_configs[0].height
+        * max_area_depth_block_configs[0].width
+    )
+    logger.info("Using block config=%s", max_area_depth_block_configs[0])
+    logger.info(
+        "Quality of the block config w.r.t. max volume block config=%s",
+        100.0 * (current_volume / largest_volume),
+    )
+    return max_area_depth_block_configs[0]
+
+
+def compress_weights(
+    weights,
+    weights_zp,
+    weights_layout,
+    ifm_bitdepth,
+    block_depth,
+    dilation,
+    accel_type,
+    is_depthwise=False,
+):
+    """Obtain compressed weights from vela

Review comment:
       Explained further.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695440184



##########
File path: tests/python/contrib/test_ethosu/test_legalize.py
##########
@@ -0,0 +1,345 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument
+
+import pytest
+
+pytest.importorskip("ethosu.vela")
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.backend.contrib import ethosu
+from tvm.relay.backend.contrib.ethosu import legalize, preprocess
+from tvm.relay.dataflow_pattern import *
+from tvm.relay.op.contrib.ethosu import *
+import relay_ir_builder
+
+
+def test_split_indices_legalize():
+    def create_graph(axis):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_relu = relay.nn.relu(x)
+        split_o = relay.split(x_relu, [5, 20, 45], axis).tuple_value
+        return relay.Function([x], split_o)
+
+    def expected_mod_axis1():
+        expected_ir_string = """
+        #[version = "0.0.5"]

Review comment:
       Thanks! at this point, if its not necessary, we'd prefer to keep it like this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-914956781


   @Mousius @ashutosh-arm , shall we prioritize to land this next as it is approved ? it is getting conflicts from CMSISNN PRs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbs-octoml commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-904956585


   (see https://discuss.tvm.apache.org/t/rfc-ethosn-arm-ethos-n-integration/6680 for RFC overview)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] jroesch commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
jroesch commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-907495303


   @mbs-octoml @manupa-arm I think the ARM folks are out on vacation over this weekend, we should try to shoot for a resolution on this part of the patch set by Tuesday-ish? I think I understand where we are going, maybe we can work to define a better C++/Python split as we go forward. It seems like this split is fine given it mostly is focused on Ethos-U vs. in the core compiler. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbs-octoml commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-908499201


   To be clear LGTM for this one modulo all the existing comments, and looks pretty good out to P5. Just:
     1) a suggestion to move some files to follow existing conventions, 
     2) put some type annots for relay types if it's not hard to do so,
     3) I expect by P4 we'l have the relay_to_tir extension point we want in place so shouldn't need to replicate the lower te pass with it's dependency on the old compile engine.
   
   For my own education I'd like to understand the custom te + custom scheduling combination being used here, some octoml folks are meeting Tue AM so I'm going to demand an explanation then!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-914464858


   @mbs-octoml I enabled the check in task_mypy.sh; I dont think there is nothing to change in mypy.ini as the global setting is "ignore_errors = False"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm edited a comment on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm edited a comment on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-910615699


   I've addressed/anwsered existing comments, I ll take pass tommorow to add type annotations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbs-octoml commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-908499201


   To be clear LGTM for this one modulo all the existing comments, and looks pretty good out to P5. Just:
     1) a suggestion to move some files to follow existing conventions, 
     2) put some type annots for relay types if it's not hard to do so,
     3) I expect by P4 we'l have the relay_to_tir extension point we want in place so shouldn't need to replicate the lower te pass with it's dependency on the old compile engine.
   
   For my own education I'd like to understand the custom te + custom scheduling combination being used here, some octoml folks are meeting Tue AM so I'm going to demand an explanation then!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-909087831


   @jroesch @mbs-octoml ,
   
   Sorry have been busy getting all the PRs up and green (which they are now :) . Yes, we were off over long-weekend here in the UK.
   
   Given its python dependency of Vela, we find keeping the integration and passes in python for the NPU lowering is beneficial, especially in debugging. However, we have moved that involves relay traversal implementations (bar the pattern rewrites -- because the python hook is just an interface and I believe the rewriting is actually happening in C++) to C++ while maintaining a python interface to cater the integration. Performance was the consideration to move the pass impl to C++. If we find some passes are slow, we'll put effort to move the impl to C++.
   
   @mbs-octoml 
   
   For 1.)
   
   > Should the op defs go in python/tvm/relay/op/contrib/ethosu.py (where you currently have just the MergeComposite pattern defs), and the te defs should go in python/tvm/topi/ethosu/conv2d.py? We've gotten this far without a contrib under python/tvm/relay/backend so trying to follow existing convention as much as possible even with the custom scheduling would be nice.
   
   We do have a src/relay/backend/contrib/ -- this is just mirroring that folder in python, therefore I think it is less confusing to create it there.
   for op definitions, @mbaret WDYT ? -- Having everything closer to the Ethos-U codegen seems easier to navigate for me.
   
   For 2.)
   
   Yes, we agree to add the type annotations for relay types.
   
   For 3.)
   
   @mbaret , we need that by P3 right?
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695434771



##########
File path: python/tvm/relay/backend/contrib/ethosu/errors.py
##########
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=super-init-not-called
+"""This module is to hold all type of errors associated Arm(R) Ethos(TM)-U NPU Codegen"""
+
+
+class EthosUCodegenError(Exception):
+    """Base class for all exceptions related to Codegen"""
+
+    def __init__(self, data):
+        self.message = "EthosUCodegenError:" + data
+
+    def __str__(self):
+        return self.message
+
+
+class UnsupportedLayout(EthosUCodegenError):
+    """Raised when unsupported layout is encountered in the codegen"""

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] NicolaLancellotti commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
NicolaLancellotti commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r696663975



##########
File path: python/tvm/relay/backend/contrib/ethosu/vela_api.py
##########
@@ -0,0 +1,313 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This is an adapter module for conversions between TVM and Vela.
+The following conversion APIs are added :
+    *Obtaining the best block config
+    *Compressing weights
+    *Packing biases
+"""
+import logging
+import math
+import numpy as np
+from ethosu.vela import api as vapi
+
+from tvm.relay.backend.contrib.ethosu import util
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("Ethos-U")
+
+VELA_TO_NP_DTYPES = {
+    vapi.NpuDataType.UINT8: np.uint8,
+    vapi.NpuDataType.UINT16: np.uint16,
+    vapi.NpuDataType.INT8: np.int8,
+    vapi.NpuDataType.INT16: np.int16,
+    vapi.NpuDataType.INT32: np.int32,
+}
+
+SCALE_BIAS_LENGTH = 10
+
+
+def get_optimal_block_config(npu_op, accel_type):
+    """
+    "The NPU's unit of work is known as a block. It will fetch block(s) from Input
+    Feature Map (IFM) and a compute block for Output Feature Map (OFM).
+    Therefore, we need to pick an optimal block configuration considering bandwidth
+    to bring IFM blocks and the number of OFM block computes need to happen
+    to cover the OFM as indicated by the npu op.
+
+    Parameters
+    ----------
+    npu_op : ethosu.vela.api.NpuOperation
+        The NPU operation and its params
+    accel_type : ethosu.vela.api.NpuAccelerator
+        The NPU accelerator variant
+    Returns

Review comment:
       ```suggestion
   
       Returns
   ```
   There has to be a blank line before Returns in order for the doc to be built correctly.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] NicolaLancellotti commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
NicolaLancellotti commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r696731249



##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU supported operators."""
+import numpy as np
+
+from tvm.relay.expr import Constant
+from tvm.relay.op.contrib.register import register_pattern_table
+from tvm.relay.dataflow_pattern import wildcard, is_op, is_constant
+from tvm.relay.backend.contrib.ethosu.util import QConv2DArgs
+from tvm.relay.backend.contrib.ethosu.util import BiasAddArgs
+from tvm.relay.backend.contrib.ethosu.util import RequantArgs
+from tvm.relay.backend.contrib.ethosu.util import get_dim_value
+from ethosu.vela import api as vapi
+
+
+def check_strides(strides):
+    """This function checks whether strides are within the limits supported by the NPU"""
+    stride_range = (1, 3)
+    smin, smax = stride_range
+    if not smax >= strides[0] >= smin:
+        return False
+    if not smax >= strides[1] >= smin:
+        return False
+    return True
+
+
+def check_valid_dtypes(tensor_params):
+    """This function checks whether dtypes are supported by the NPU"""
+    supported_dtypes = (np.uint8, np.int8)
+    for tep in tensor_params:
+        # Check for dtypes
+        if np.dtype(tep.dtype) not in supported_dtypes:
+            return False
+        # Check for shape sizes
+        if any(dimlen > 65536 for dimlen in tep.shape):
+            return False
+    return True
+
+
+def check_weights(weights, dilation):
+    """This function checks whether weight tensor is compatible with the NPU"""
+    dilated_height_range = (1, 64)
+    dilated_hxw_range = (1, 64 * 64)
+    weights_limit = 127 * 65536
+    dilated_width = (weights.shape[get_dim_value(weights.layout, "W")] - 1) * dilation[0] + 1
+    dilated_height = (weights.shape[get_dim_value(weights.layout, "H")] - 1) * dilation[1] + 1
+    dh_min, dh_max = dilated_height_range
+    if not dh_min <= dilated_height <= dh_max:
+        return False
+    dilated_hxw = dilated_height * dilated_width
+    dhxw_min, dhxw_max = dilated_hxw_range
+    if not dhxw_min <= dilated_hxw <= dhxw_max:
+        return False
+    # A saturation upper bound check for accumulators
+    weights.values = weights.values - weights.q_params.zero_point
+    axis = (
+        get_dim_value(weights.layout, "H"),
+        get_dim_value(weights.layout, "W"),
+        get_dim_value(weights.layout, "I"),
+    )
+    sum_weights = np.amax(np.sum(np.absolute(weights.values), axis=axis))
+    if not sum_weights <= weights_limit:
+        return False
+    return True
+
+
+def check_bias(bias):
+    """This function checks whether the bias values fit in 40 bits"""
+    if bias and bias.dtype == np.dtype("int64"):
+        valid = all(len(bin(bias_value)[2:]) <= 40 for bias_value in bias.values)
+        return valid
+    return True
+
+
+def check_batch_size(ifm):
+    """This function checks for the number of batches vela currently supports"""
+    if ifm.shape[0] != 1:
+        return False
+    return True
+
+
+def check_dilation(dilation):
+    """This function checks whether dilation is within the limits supported by the NPU"""
+    dilation_range = (1, 2)
+    dmin, dmax = dilation_range
+    if not dmin <= dilation[0] <= dmax:
+        return False
+    if not dmin <= dilation[1] <= dmax:
+        return False
+    return True
+
+
+def check_padding(padding, bounds):
+    """This function checks whether padding is within the limits supported by the NPU"""
+    if len(padding) != 4 or len(bounds) != 4:
+        return False
+    top, left, bottom, right = padding
+    topb, leftb, bottomb, rightb = bounds
+    if top > topb or left > leftb or bottom > bottomb or right > rightb:
+        return False
+    return True
+
+
+class TensorParams:
+    """
+    This class will parse a tvm Expr along with quantization scale
+    and zero point to populate parameters that are required
+    for the creation of tensors in Vela.
+    """
+
+    def __init__(self, tensor, layout=None, scale=None, zero_point=None):
+        self.tensor = tensor
+        if isinstance(tensor, Constant):
+            self.values = tensor.data.asnumpy()
+        else:
+            self.values = None
+        self.dtype = tensor.checked_type.dtype
+        self.shape = [int(i) for i in tensor.checked_type.shape]
+        self.layout = layout
+
+        if scale is not None and zero_point is not None:
+            self.q_params = vapi.NpuQuantization(
+                scale.data.asnumpy().astype("float32"), zero_point.data.asnumpy().astype(self.dtype)
+            )
+        else:
+            # put default values
+            self.q_params = vapi.NpuQuantization(1.0, 0)
+
+
+class QnnConv2DParams:
+    """
+    This class will parse a Call to a ethosu.qnn_conv2d composite function
+    and extract quantization information of all the associated tensors.
+    """
+
+    composite_name = "ethosu.qnn_conv2d"
+    # The NPU only supports padding upto the numbers as follows
+    padding_bounds = [31, 31, 32, 32]
+    activation_map = {"clip": "CLIP"}
+
+    def __init__(self, func_body):
+        activation = None
+        if str(func_body.op) in self.activation_map.keys():
+            activation = func_body
+            requantize_op = activation.args[0]
+        else:
+            requantize_op = func_body
+        bias_add = requantize_op.args[0]
+        qnn_conv2d = bias_add.args[0]
+        data_layout = qnn_conv2d.attrs.data_layout
+        kernel_layout = qnn_conv2d.attrs.kernel_layout
+        # We consider the weights & biases as params as it should be a Constant
+        self.weights = TensorParams(
+            qnn_conv2d.args[QConv2DArgs.WEIGHTS.value],
+            kernel_layout,
+            qnn_conv2d.args[QConv2DArgs.WEIGHTS_SCALE.value],
+            qnn_conv2d.args[QConv2DArgs.WEIGHTS_ZERO_POINT.value],
+        )
+
+        self.biases = TensorParams(
+            bias_add.args[BiasAddArgs.BIASES.value],
+            data_layout,
+            requantize_op.args[RequantArgs.IFM_SCALE.value],
+            requantize_op.args[RequantArgs.IFM_ZERO_POINT.value],
+        )
+        self.ifm = TensorParams(
+            qnn_conv2d.args[QConv2DArgs.IFM.value],
+            data_layout,
+            qnn_conv2d.args[QConv2DArgs.IFM_SCALE.value],
+            qnn_conv2d.args[QConv2DArgs.IFM_ZERO_POINT.value],
+        )
+        self.ofm = TensorParams(
+            func_body,
+            data_layout,
+            requantize_op.args[RequantArgs.OFM_SCALE.value],
+            requantize_op.args[RequantArgs.OFM_ZERO_POINT.value],
+        )
+        self.padding = qnn_conv2d.attrs.padding
+        self.strides = qnn_conv2d.attrs.strides
+        self.dilation = qnn_conv2d.attrs.dilation
+        self.activation = activation
+
+        # If groups are equal to channel, its a depthwise_conv2d
+        self.groups = qnn_conv2d.attrs.groups
+        self.is_depthwise = False
+        channels_axis = {"HWIO": 3, "HWOI": 2}
+        if qnn_conv2d.attrs.groups == self.weights.shape[channels_axis[kernel_layout]]:
+            self.is_depthwise = True
+
+    def is_valid(self):
+        """
+        This function checks whether QnnConv2D has compatible attributes with the NPU
+        """
+        tensor_params = [self.weights, self.ifm, self.ofm]
+        if not check_valid_dtypes(tensor_params):
+            return False
+        if not check_weights(self.weights, self.dilation):
+            return False
+        if not check_bias(self.biases):
+            return False
+        if not check_strides(self.strides):
+            return False
+        if not check_batch_size(self.ifm):
+            return False
+        if not check_dilation(self.dilation):
+            return False
+        if not check_padding(self.padding, self.padding_bounds):
+            return False
+        legal_groups = [1, self.ofm.shape[3]]
+        if self.groups not in legal_groups:
+            return False
+        # This should be a valid QnnDepthwise2DParams, not QnnConv2DParams
+        if self.is_depthwise:
+            return False
+        return True
+
+
+def qnn_conv2d_pattern():
+    """
+    This function creates the pattern for qnn.conv2D with optional fused RELU activation.
+    """
+    qnn_conv2d = is_op("qnn.conv2d")(
+        wildcard(), is_constant(), is_constant(), is_constant(), is_constant(), is_constant()
+    ).has_attr({"kernel_layout": "HWIO"})
+    bias_add = is_op("nn.bias_add")(qnn_conv2d, is_constant())
+    req = is_op("qnn.requantize")(
+        qnn_conv2d | bias_add, is_constant(), is_constant(), is_constant(), is_constant()
+    )
+    clip_or_req = req.optional(is_op("clip"))
+    return clip_or_req
+
+
+@register_pattern_table("ethosu")
+def pattern_table():
+    return [
+        ("ethosu.qnn_conv2d", qnn_conv2d_pattern(), lambda pat: QnnConv2DParams(pat).is_valid())

Review comment:
       ```suggestion
           (QnnConv2DParams.composite_name, qnn_conv2d_pattern(), lambda pat: QnnConv2DParams(pat).is_valid())
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695438902



##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU supported operators."""
+import numpy as np
+
+from tvm.relay.expr import Constant
+from tvm.relay.op.contrib.register import register_pattern_table
+from tvm.relay.dataflow_pattern import wildcard, is_op, is_constant
+from tvm.relay.backend.contrib.ethosu.util import QConv2DArgs
+from tvm.relay.backend.contrib.ethosu.util import BiasAddArgs
+from tvm.relay.backend.contrib.ethosu.util import RequantArgs
+from tvm.relay.backend.contrib.ethosu.util import get_dim_value
+from ethosu.vela import api as vapi
+
+
+def check_strides(strides):
+    """Checks whether strides are within the limits supported by the hardware"""

Review comment:
       I used NPU, instead -- I think its a reasonable ask to know what file the reader is reading.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] NicolaLancellotti commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
NicolaLancellotti commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r696641990



##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,183 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out code generation.
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enum to obtain the correct index
+    of qnn.conv2d arguments.
+    """
+
+    IFM = 0
+    WEIGHTS = 1
+    IFM_ZERO_POINT = 2
+    WEIGHTS_ZERO_POINT = 3
+    IFM_SCALE = 4
+    WEIGHTS_SCALE = 5
+
+
+class RequantArgs(Enum):
+    """
+    This is a helper enum to obtain the correct index
+    of qnn.requantize arguments.
+    """
+
+    IFM_SCALE = 1
+    IFM_ZERO_POINT = 2
+    OFM_SCALE = 3
+    OFM_ZERO_POINT = 4
+
+
+class BiasAddArgs(Enum):
+    """
+    This is a helper enums to obtain the correct index
+    of qnn.bias_add arguments.
+    """
+
+    BIASES = 1
+
+
+class ClipArgs(Enum):
+    """
+    This is a helper enums to obtain the correct index
+    of clip arguments.
+    """
+
+    A_MIN = 1
+    A_MAX = 2
+
+
+def is_composite_func(func, name):
+    """
+    This method checks whether the call is to
+    a composite function of a given name.
+
+    Parameters
+    ----------
+    func : relay.Function
+        The header to be displayed along with the dump.
+
+    name : str
+        The candidate name to be checked
+
+    Returns
+    --------
+    a boolean
+    """
+
+    if not hasattr(func, "attrs"):
+        return False
+    if "Composite" not in func.attrs.keys():
+        return False
+    composite_name = func.attrs["Composite"]
+
+    if composite_name != name:
+        return False
+    return True
+
+
+def get_range_for_dtype_str(dtype):
+    """
+    Produce the min,max for a give data type.
+
+    Parameters
+    ----------
+    dtype : str
+        a type string (e.g., int8)
+
+    Returns
+    -------
+    type_info.min : int
+        the minimum of the range
+    type_info.max : int
+        the maximum of the range
+    """
+
+    try:
+        type_info = np.iinfo(dtype)
+    except ValueError:
+        type_info = np.finfo(dtype)
+    return type_info.min, type_info.max
+
+
+def round_away_zero(f):
+    """round the number away from zero towards +inf / -inf"""
+    offset = -0.5 if (f < 0) else 0.5
+    return np.trunc(f + offset)
+
+
+def round_up(a, b):
+    """round up to a multiple of b"""

Review comment:
       ```suggestion
       """Round up to a multiple of b"""
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695437635



##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index

Review comment:
       Done

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn conv2d arguments
+    """
+
+    ifm = 0
+    weights = 1
+    ifm_zero_point = 2
+    weights_zero_point = 3
+    ifm_scale = 4
+    weights_scale = 5
+
+
+class RequantArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn requantize arguments
+    """
+
+    ifm_scale = 1
+    ifm_zero_point = 2
+    ofm_scale = 3
+    ofm_zero_point = 4
+
+
+class BiasAddArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    biases = 1
+
+
+class ClipArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    a_min = 1
+    a_max = 2
+
+
+class MaxPoolArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm = 0
+
+
+class AddArgs(Enum):
+    """This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm0 = 0
+    ifm1 = 1
+    ifm0_scale = 2
+    ifm0_zero_point = 3
+    ifm1_scale = 4
+    ifm1_zero_point = 5
+    ofm_scale = 6
+    ofm_zero_point = 7
+
+
+def is_composite_func(func, name):
+    """
+    This a method to check whether the call is to

Review comment:
       Done

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn conv2d arguments
+    """
+
+    ifm = 0
+    weights = 1
+    ifm_zero_point = 2
+    weights_zero_point = 3
+    ifm_scale = 4
+    weights_scale = 5
+
+
+class RequantArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn requantize arguments
+    """
+
+    ifm_scale = 1
+    ifm_zero_point = 2
+    ofm_scale = 3
+    ofm_zero_point = 4
+
+
+class BiasAddArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    biases = 1
+
+
+class ClipArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn bias_add arguments
+    """
+
+    a_min = 1
+    a_max = 2
+
+
+class MaxPoolArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm = 0
+
+
+class AddArgs(Enum):
+    """This is a helper enums to access the correct index
+    max pool arguments
+    """
+
+    ifm0 = 0
+    ifm1 = 1
+    ifm0_scale = 2
+    ifm0_zero_point = 3
+    ifm1_scale = 4
+    ifm1_zero_point = 5
+    ofm_scale = 6
+    ofm_zero_point = 7
+
+
+def is_composite_func(func, name):
+    """
+    This a method to check whether the call is to
+    a composite function of the "name".

Review comment:
       Done ( I think)

##########
File path: python/tvm/relay/backend/contrib/ethosu/vela_api.py
##########
@@ -0,0 +1,314 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+conversions between TVM and Vela. Therefore, all interactions with the

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] NicolaLancellotti commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
NicolaLancellotti commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r696745633



##########
File path: src/relay/op/contrib/ethosu/convolution.cc
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/op/contrib/ethosu/convolution.cc
+ * \brief Property def of the Arm(R) Ethos(TM)-U NPU convolution ops.
+ */
+#include "../../nn/convolution.h"
+
+#include <tvm/relay/base.h>
+#include <tvm/relay/op.h>
+#include <tvm/relay/qnn/attrs.h>
+#include <tvm/tir/analysis.h>
+#include <tvm/tir/data_layout.h>
+
+#include "../../../qnn/utils.h"
+#include "common.h"
+
+namespace tvm {
+namespace relay {
+namespace op {
+namespace contrib {
+namespace ethosu {
+
+/*! \brief Attributes used by the Ethos(TM)-U NPU convolution operator */
+struct EthosuConv2DAttrs : public tvm::AttrsNode<EthosuConv2DAttrs> {
+  double ifm_scale;
+  int ifm_zero_point;
+  int weight_zero_point;
+  double ofm_scale;
+  int ofm_zero_point;
+  Array<IndexExpr> kernel_shape;
+  IndexExpr ofm_channels;
+  Array<IndexExpr> strides;
+  Array<IndexExpr> padding;
+  Array<IndexExpr> dilation;
+  String activation;
+  int clip_min;
+  int clip_max;
+  String upscale;
+  tvm::String ifm_layout;
+  tvm::String ofm_layout;
+
+  TVM_DECLARE_ATTRS(EthosuConv2DAttrs, "relay.attrs.EthosuConv2DAttrs") {
+    TVM_ATTR_FIELD(ifm_scale).describe("The quantization scale for the Input Feature Map tensor.");
+    TVM_ATTR_FIELD(ifm_zero_point)
+        .describe("The quantization zero point for the Input Feature Map tensor.");
+    TVM_ATTR_FIELD(weight_zero_point)
+        .describe("The quantization zero point for the weight tensor.");
+    TVM_ATTR_FIELD(ofm_scale).describe("The quantization scale for the Output Feature Map tensor.");
+    TVM_ATTR_FIELD(ofm_zero_point)
+        .describe("The quantization zero point for the Output Feature Map tensor.");
+    TVM_ATTR_FIELD(kernel_shape)
+        .describe("The 2 dimensional kernel shape as (kernel_height, kernel_width).")
+        .set_default(NullValue<Array<IndexExpr> >());
+    TVM_ATTR_FIELD(ofm_channels)
+        .describe("The number of OFM channels.")
+        .set_default(NullValue<IndexExpr>());
+    TVM_ATTR_FIELD(strides)
+        .set_default(Array<IndexExpr>({1, 1}))
+        .describe("The 2 dimensional strides as (stride_height, stride_width).");
+    TVM_ATTR_FIELD(padding)
+        .set_default(Array<IndexExpr>({0, 0, 0, 0}))
+        .describe("The 4 dimensional padding as (pad_top, pad_left, pad_bottom, pad_right).");
+    TVM_ATTR_FIELD(dilation)
+        .set_default(Array<IndexExpr>({1, 1}))
+        .describe("The 2 dimensional dilation as (dilation_height, dilation_width).");
+    TVM_ATTR_FIELD(activation)
+        .describe(
+            "The activation function to use. "
+            "'NONE' - no activation function. "
+            "'CLIP' - clip the output between clip_min and clip_max. "
+            "'TANH' - tanh activation function. "
+            "'SIGMOID' - sigmoid activation function. "
+            "'LUT' - use a look-up table to perform the activation function.")
+        .set_default("NONE");
+    TVM_ATTR_FIELD(clip_min)
+        .describe("The minimum clipping value if activation = 'CLIP'.")
+        .set_default(0);
+    TVM_ATTR_FIELD(clip_max)
+        .describe("The maximum clipping value if activation = 'CLIP'.")
+        .set_default(0);
+    TVM_ATTR_FIELD(upscale)
+        .describe(
+            "The 2x2 upscaling mode to apply to the Input Feature Map tensor. "
+            "'NONE' - no upscaling. "
+            "'NEAREST' - upscale using nearest neighbour. "
+            "'ZEROS' - upscale using zeros.")
+        .set_default("NONE");
+    TVM_ATTR_FIELD(ifm_layout)
+        .set_default("NHWC")
+        .describe("The layout of the Input Feature Map tensor. Can be 'NHWC' or 'NHCWB16'.");
+    TVM_ATTR_FIELD(ofm_layout)
+        .set_default("NHWC")
+        .describe("The layout of the Output Feature Map tensor. Can be 'NHWC' or 'NHCWB16'.");
+  }
+};
+
+TVM_REGISTER_NODE_TYPE(EthosuConv2DAttrs);
+
+bool EthosuConv2DRel(const Array<Type>& types, int num_inputs, const Attrs& attrs,
+                     const TypeReporter& reporter) {
+  CHECK_EQ(types.size(), 5);
+  const auto* ifm = types[0].as<TensorTypeNode>();
+  const auto* weight = types[1].as<TensorTypeNode>();
+  const auto* scale_bias = types[2].as<TensorTypeNode>();
+  if (ifm == nullptr || weight == nullptr) return false;
+  const auto* param = attrs.as<EthosuConv2DAttrs>();
+  CHECK(param != nullptr) << "EthosuConv2DAttrs cannot be nullptr.";
+  CHECK(ifm->dtype == DataType::UInt(8) || ifm->dtype == DataType::Int(8))
+      << "Expected ethosu_conv2d type(uint8) or type(int8) for ifm but was " << ifm->dtype;
+  CHECK(weight->dtype == DataType::UInt(8) || weight->dtype == DataType::Int(8))
+      << "Expected ethosu_conv2d type(uint8) or type(int8) for weight but was " << weight->dtype;
+  CHECK(scale_bias->dtype == DataType::UInt(8))
+      << "Expected ethosu_conv2d type(uint8) for scale_bias but was " << scale_bias->dtype;
+
+  // The scale_bias should be provided as a tensor of size {ofm_channels, 10}
+  reporter->Assign(types[2], TensorType({weight->shape[0], 10}, DataType::UInt(8)));
+
+  // Assign weight type {ofm_channels, kernel_height, kernel_width, ifm_channels}
+  reporter->Assign(types[1], TensorType({param->ofm_channels, param->kernel_shape[0],
+                                         param->kernel_shape[1], weight->shape[3]},
+                                        weight->dtype));
+
+  // Assign ofm type
+  auto ofm_shape =
+      EthosuInferKernelOutput(ifm->shape, param->ifm_layout, param->ofm_layout, param->kernel_shape,
+                              param->ofm_channels, param->dilation, param->strides, param->padding);
+  reporter->Assign(types[4], TensorType(ofm_shape, ifm->dtype));
+  return true;
+}
+
+Expr MakeEthosuConv2D(Expr ifm, Expr weight, Expr scale_bias, Expr lut, double ifm_scale,
+                      int ifm_zero_point, int weight_zero_point, double ofm_scale,
+                      int ofm_zero_point, Array<IndexExpr> kernel_shape, IndexExpr ofm_channels,
+                      Array<IndexExpr> strides, Array<IndexExpr> padding, Array<IndexExpr> dilation,
+                      String activation, int clip_min, int clip_max, String upscale,
+                      String ifm_layout, String ofm_layout) {
+  auto attrs = make_object<EthosuConv2DAttrs>();
+  attrs->ifm_scale = ifm_scale;
+  attrs->ifm_zero_point = ifm_zero_point;
+  attrs->weight_zero_point = weight_zero_point;
+  attrs->ofm_scale = ofm_scale;
+  attrs->ofm_zero_point = ofm_zero_point;
+  attrs->kernel_shape = std::move(kernel_shape);
+  attrs->ofm_channels = std::move(ofm_channels);
+  attrs->strides = std::move(strides);
+  attrs->padding = std::move(padding);
+  attrs->dilation = std::move(dilation);
+  attrs->activation = std::move(activation);
+  attrs->clip_min = clip_min;
+  attrs->clip_max = clip_max;
+  attrs->upscale = std::move(upscale);
+  attrs->ifm_layout = std::move(ifm_layout);
+  attrs->ofm_layout = std::move(ofm_layout);
+  static const Op& op = Op::Get("contrib.ethosu.conv2d");
+  return Call(op, {ifm, weight, scale_bias, lut}, Attrs(attrs), {});
+}
+
+TVM_REGISTER_GLOBAL("relay.op._make.ethosu_conv2d").set_body_typed(MakeEthosuConv2D);
+
+RELAY_REGISTER_OP("contrib.ethosu.conv2d")
+    .describe(R"code(Arm(R) Ethos(TM)-U NPU 2D quantized convolution operator.
+
+This Relay operator corresponds to the hardware-implemented quantized
+convolution operation found on Ethos(TM)-U NPUs. It accepts either NHWC
+or NHCWB16 format for the input data (input feature map, or IFM) and

Review comment:
       ```suggestion
   or NHCWB16 format for the input data (Input Feature Map, or IFM) and
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695437428



##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r700500936



##########
File path: tests/python/contrib/test_ethosu/test_legalize.py
##########
@@ -0,0 +1,333 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument
+
+import pytest
+
+pytest.importorskip("ethosu.vela")
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.backend.contrib import ethosu
+from tvm.relay.backend.contrib.ethosu import legalize, preprocess
+from tvm.relay.dataflow_pattern import *
+from tvm.relay.op.contrib.ethosu import *
+import relay_ir_builder
+
+
+def test_split_indices_legalize():
+    def create_graph(axis):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_relu = relay.nn.relu(x)
+        split_o = relay.split(x_relu, [5, 20, 45], axis).tuple_value

Review comment:
       just a variable name to say output of split ?

##########
File path: tests/python/contrib/test_ethosu/test_legalize.py
##########
@@ -0,0 +1,333 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument
+
+import pytest
+
+pytest.importorskip("ethosu.vela")
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.backend.contrib import ethosu
+from tvm.relay.backend.contrib.ethosu import legalize, preprocess
+from tvm.relay.dataflow_pattern import *
+from tvm.relay.op.contrib.ethosu import *
+import relay_ir_builder
+
+
+def test_split_indices_legalize():
+    def create_graph(axis):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_relu = relay.nn.relu(x)
+        split_o = relay.split(x_relu, [5, 20, 45], axis).tuple_value
+        return relay.Function([x], split_o)
+
+    def expected_mod_axis1():
+        expected_ir_string = """
+        #[version = "0.0.5"]
+        def @tvmgen_default_ethosu_main_0(%x: Tensor[(1, 50, 50, 3), float32]) -> (Tensor[(1, 5, 50, 3), float32],\
+                                                               Tensor[(1, 15, 50, 3), float32],\
+                                                               Tensor[(1, 25, 50, 3), float32],\
+                                                               Tensor[(1, 5, 50, 3), float32]) {
+          %0 = nn.relu(%x) /* ty=Tensor[(1, 50, 50, 3), float32] */;
+          %1 = strided_slice(%0, begin=[0, 0, 0, 0], end=[1, 5, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 5, 50, 3), float32] */;
+          %2 = strided_slice(%0, begin=[0, 5, 0, 0], end=[1, 20, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 15, 50, 3), float32] */;
+          %3 = strided_slice(%0, begin=[0, 20, 0, 0], end=[1, 45, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 25, 50, 3), float32] */;
+          %4 = strided_slice(%0, begin=[0, 45, 0, 0], end=[1, 50, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 5, 50, 3), float32] */;
+          (%1, %2, %3, %4)
+        }
+        """
+        return tvm.parser.fromtext(expected_ir_string)
+
+    def expected_mod_axis2():
+        expected_ir_string = """
+        #[version = "0.0.5"]
+        def @tvmgen_default_ethosu_main_0(%x: Tensor[(1, 50, 50, 3), float32]) -> (Tensor[(1, 50, 5, 3), float32],\
+                                                               Tensor[(1, 50, 15, 3), float32],\
+                                                               Tensor[(1, 50, 25, 3), float32],\
+                                                               Tensor[(1, 50, 5, 3), float32]) {
+          %0 = nn.relu(%x) /* ty=Tensor[(1, 50, 50, 3), float32] */;
+          %1 = strided_slice(%0, begin=[0, 0, 0, 0], end=[1, 50, 5, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 5, 3), float32] */;
+          %2 = strided_slice(%0, begin=[0, 0, 5, 0], end=[1, 50, 20, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 15, 3), float32] */;
+          %3 = strided_slice(%0, begin=[0, 0, 20, 0], end=[1, 50, 45, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 25, 3), float32] */;
+          %4 = strided_slice(%0, begin=[0, 0, 45, 0], end=[1, 50, 50, 3], strides=[1], axes=None)\
+           /* ty=Tensor[(1, 50, 5, 3), float32] */;
+          (%1, %2, %3, %4)
+        }
+        """
+        return tvm.parser.fromtext(expected_ir_string)
+
+    mod_axis1 = tvm.IRModule()
+    mod_axis1["tvmgen_default_ethosu_main_0"] = create_graph(1)
+    mod_axis1 = legalize.LegalizeSplit()(mod_axis1)
+    expected_axis1 = expected_mod_axis1()
+    tvm.ir.assert_structural_equal(mod_axis1, expected_axis1)
+
+    mod_axis2 = tvm.IRModule()
+    mod_axis2["tvmgen_default_ethosu_main_0"] = create_graph(2)
+    mod_axis2 = legalize.LegalizeSplit()(mod_axis2)
+    expected_axis2 = expected_mod_axis2()
+    tvm.ir.assert_structural_equal(mod_axis2, expected_axis2)
+
+
+def test_split_sections_legalize():
+    def create_graph(axis, sections):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_abs = relay.abs(x)
+        split_o = relay.split(x_abs, sections, axis).tuple_value

Review comment:
       just a variable name to say output of split ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-912903328


   @jroesch @mbs-octoml PTAL when you have some time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-912551828


   > We should be able to expose it, can you clearly comment where you the exact API so Mark and I can look into replacing in parallel?
   
   @jroesch You can take a look here: https://github.com/apache/tvm/pull/8806/files#diff-ba18206383e6627c682094ad5b1a211be9dd8315e82b9c5f970496ce57ebe48c (from the next PR). That's the custom Relay -> TE lowering function we're making use of which demonstrates the API.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbs-octoml commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-906640059


   On the py/cc split: I see the vela deps are mostly for instruction stream encoding and help checking dimenions etc are supported. Theoretically they could be ffi'd back into tvm. But even without that python/relay/backend/contrib/ethosu/tir/...,  python/relay/backend/contraib/ethosu/codegen etc could be in cc, but can appreciate the convenience of staying mostly in py. As a compromise could you add type annots to the py code, particularly on callbacks reaching into tvm IR nodes? That would give me confidence we can spot when changes on the cc side would impact the py side. 
   
   I left a comment on I think PR4 -- I've checked out and build PR6 which I'm using to see the big picture, so will sprinkle comments ahead as I go. I'm out of my depth for a few sections though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-905023897


   Hi Mark. We intend to upstream 5 PRs for this work (6 in total but one is already merged), check out the tracking issue here: https://github.com/apache/tvm/issues/8482. We currently have up to 4/6 upstream (see here: https://github.com/apache/tvm/pull/8811). Thanks to GitHub seemingly not supporting incremental patch chains, each subsequent PR also includes all the stuff before it (i.e. PR 4 contains the work from PR 3 and PR 2 as well). If you want a more 'complete' view, you can take a look therefore at the later PRs. It's out intention to have PRs 5 and 6 up by the end of this week. By PR 5, you should be able to run a convolution on the NPU model that we've contributed to CI (in PR 1).
   
   Also, just a correction on the RFC you linked. That's for Ethos-N, a different class of NPU that we have also added support for in TVM (mostly last year). The RFC for Ethos-U is here: https://github.com/apache/tvm-rfcs/pull/11. I'll look into updating our PRs to clearly link to both the tracking issue and RFC.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695438548



##########
File path: python/tvm/relay/backend/contrib/ethosu/vela_api.py
##########
@@ -0,0 +1,314 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+conversions between TVM and Vela. Therefore, all interactions with the
+Vela API are supposed to go through this adapter, with the hope that
+any changes to Vela API, TVM only needs to change this file.
+The following conversion APIs are added :
+    *Obtaining the best block config
+    *Compressing weights
+    *Packing biases
+"""
+import logging
+import math
+import numpy as np
+from ethosu.vela import api as vapi
+
+from tvm.relay.backend.contrib.ethosu import util
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("Ethos-U")
+
+VELA_TO_NP_DTYPES = {
+    vapi.NpuDataType.UINT8: np.uint8,
+    vapi.NpuDataType.UINT16: np.uint16,
+    vapi.NpuDataType.INT8: np.int8,
+    vapi.NpuDataType.INT16: np.int16,
+    vapi.NpuDataType.INT32: np.int32,
+}
+
+SCALE_BIAS_LENGTH = 10
+
+
+def get_optimal_block_config(npu_op, accel_type):
+    """
+    "The NPU's unit of work is known as a block. It will fetch block(s) from Input
+    Feature Map (IFM) and a compute block for Output Feature Map (OFM).
+    Therefore, we need to pick an optimal block configuration considering bandwidth
+    to bring IFM blocks and the number of OFM block computes need to happen
+    to cover the OFM as indicated by the npu op.
+
+    Parameters
+    ----------
+    npu_op : ethosu.vela.api.NpuOperation
+        The NPU operation and its params
+    accel_type : ethosu.vela.api.NpuAccelerator
+        The NPU accelerator variant
+    Returns
+    -------
+    ethosu.vela.api.NpuShape3d :
+        The optimal block config for the operator
+    """
+    all_valid_block_configs = vapi.npu_find_block_configs(npu_op, accel_type)
+    return _get_optimal_block_config(all_valid_block_configs)
+
+
+def _get_optimal_block_config(all_valid_block_configs):
+    """An internal function to get block config with largest depth
+    and then highest volume/area"""
+    assert isinstance(all_valid_block_configs, list)
+    for block_cfg in all_valid_block_configs:
+        assert isinstance(block_cfg, vapi.NpuShape3D)
+
+    # Getting the largest volume block for benchmarksing
+    all_valid_block_configs.sort(
+        key=lambda _cfg: _cfg.depth * _cfg.height * _cfg.width, reverse=True
+    )
+    largest_volume_block_config = all_valid_block_configs[0]
+    largest_volume = (
+        largest_volume_block_config.depth
+        * largest_volume_block_config.height
+        * largest_volume_block_config.width
+    )
+
+    all_valid_block_configs.sort(key=lambda _cfg: _cfg.depth, reverse=True)
+    max_d = all_valid_block_configs[0].depth
+    max_depth_block_configs = [_cfg for _cfg in all_valid_block_configs if _cfg.depth == max_d]
+    max_depth_block_configs.sort(key=lambda _cfg: _cfg.height * _cfg.width, reverse=True)
+    max_area = max_depth_block_configs[0].height * max_depth_block_configs[0].width
+    max_area_depth_block_configs = [
+        _cfg for _cfg in max_depth_block_configs if _cfg.height * _cfg.width == max_area
+    ]
+    # This to get a deterministic anwser everytime
+    max_area_depth_block_configs.sort(key=lambda _cfg: _cfg.height, reverse=True)
+    assert len(max_area_depth_block_configs) > 0
+    current_volume = (
+        max_area_depth_block_configs[0].depth
+        * max_area_depth_block_configs[0].height
+        * max_area_depth_block_configs[0].width
+    )
+    logger.info("Using block config=%s", max_area_depth_block_configs[0])
+    logger.info(
+        "Quality of the block config w.r.t. max volume block config=%s",
+        100.0 * (current_volume / largest_volume),
+    )
+    return max_area_depth_block_configs[0]
+
+
+def compress_weights(
+    weights,
+    weights_zp,
+    weights_layout,
+    ifm_bitdepth,
+    block_depth,
+    dilation,
+    accel_type,
+    is_depthwise=False,
+):
+    """Obtain compressed weights from vela
+
+    Parameters
+    ----------
+    weights : numpy.ndarray
+        The raw weights
+    weights_zp : int
+        The zero point of the weights
+    weights_layout : str
+        A string literal indicating the layout
+        Supported values : HWIO, HWOI, OHWI
+    ifm_bitdepth : int
+        The bit depth of the ifm the weights are used with
+    block_depth : int
+        The depth of the optimal block config for the operator
+    dilation : tuple
+        A tuple of 2 elements indicating dilation in h and w
+    accel_type : ethosu.vela.api.NpuAccelerator
+        The NPU accelerator variant
+    is_depthwise : bool, Optional
+        This indicates whether the weights are compressed for depthwise convolution
+
+    Returns
+    -------
+    compressed_weights : bytearray
+        Compressed weights
+    """
+    layout_transform_indices = {"HWIO": (3, 0, 1, 2), "HWOI": (2, 0, 1, 3), "OHWI": (0, 1, 2, 3)}
+    assert weights_layout in layout_transform_indices.keys()
+    assert isinstance(weights_zp, np.int64)
+    weights = weights.astype(np.int64) - weights_zp
+    # Vela needs the weights in OHWI layout
+    weights_ohwi = np.transpose(weights, layout_transform_indices[weights_layout])
+    shape_ohwi = [
+        weights.shape[layout_transform_indices[weights_layout][0]],
+        weights.shape[layout_transform_indices[weights_layout][1]],
+        weights.shape[layout_transform_indices[weights_layout][2]],
+        weights.shape[layout_transform_indices[weights_layout][3]],
+    ]
+    block_traversal = calculate_block_traversal_mode(is_depthwise, shape_ohwi, ifm_bitdepth)
+    compressed_weights = vapi.npu_encode_weights(
+        accelerator=accel_type,
+        weights_volume=weights_ohwi,
+        dilation_xy=dilation,
+        ifm_bitdepth=ifm_bitdepth,
+        ofm_block_depth=block_depth,
+        is_depthwise=is_depthwise,
+        block_traversal=block_traversal,
+    )
+    return compressed_weights
+
+
+def calculate_block_traversal_mode(is_depthwise, weights_shape_ohwi, ifm_bitdepth):
+    """Calculate a block traversal mode given whether the op is depthwise convolution,
+    shape of weights and bit-depth of the ifm.
+    """
+
+    if is_depthwise:
+        return vapi.NpuBlockTraversal.DEPTH_FIRST
+    # Determine which block traversal strategy has better DPU utilization
+    kernel_size = weights_shape_ohwi[1] * weights_shape_ohwi[2]
+    depth_utilization = weights_shape_ohwi[3] / util.round_up(
+        weights_shape_ohwi[3], 32 if ifm_bitdepth == 8 else 16
+    )
+    part_kernel_utilization = (weights_shape_ohwi[3] / util.round_up(weights_shape_ohwi[3], 8)) * (
+        kernel_size / util.round_up(kernel_size, 4 if ifm_bitdepth == 8 else 2)
+    )
+    if part_kernel_utilization >= depth_utilization or weights_shape_ohwi[3] <= 8:
+        # Part-kernel first is always better for ifm depths <= 8
+        return vapi.NpuBlockTraversal.PART_KERNEL_FIRST
+    return vapi.NpuBlockTraversal.DEPTH_FIRST
+
+
+def pack_biases(
+    biases,
+    ifm_scale,
+    ifm_dtype,
+    weight_scales,
+    ofm_scale,
+    is_activation_tanh_or_sigmoid=False,
+):
+    """
+    Obtain packed bias bytearray as the hardware requires from
+    Vela.
+    Parameters
+    ----------
+    biases : numpy.ndarray
+        The values of biases
+    ifm_scale : float
+        The quantization scale parameter of input feature map
+    ifm_dtype : numpy.dtype
+        The data type of input feature map data.
+    weight_scales : numpy.ndarray
+        The quantization scale parameter of weight feature map
+        This could be a tuple if per-channel quantization is present.
+    ofm_scale : float
+        The quantization scale parameter of output feature map.
+    is_activation_tanh_or_sigmoid : bool
+        Indicates whether the fused activation function is tanh or sigmoid.
+
+    Returns
+    -------
+    scale_bias : numpy.ndarray
+        Packed scales/biases as the hardware requires them.
+    """
+    # The BYOC infra should not partition anything else.
+    supported_ifm_dtypes = (np.uint8, np.int8, np.int16)
+    assert ifm_dtype in supported_ifm_dtypes
+
+    if weight_scales.size == 1:
+        weight_scales = [weight_scales] * biases.size
+
+    hw_bias_scales = _calculate_hw_bias_scales(
+        ifm_scale, weight_scales, ofm_scale, ifm_dtype, is_activation_tanh_or_sigmoid
+    )
+    assert len(hw_bias_scales) == biases.size
+    biases = biases.astype("int64")
+    packed_biases = bytearray()
+    for idx, scale in enumerate(hw_bias_scales):
+        packed_biases.extend(vapi.npu_encode_bias(biases[idx], *scale))
+    # Align to 16
+    # remainder = (len(packed_biases)) % 16
+    # if remainder > 0:
+    #     packed_biases.extend(bytearray(16 - remainder))
+    scale_bias = np.frombuffer(packed_biases, dtype=np.uint8)
+    scale_bias = np.reshape(scale_bias, (-1, 10))
+    return scale_bias
+
+
+def _quantize_scale(scale):
+    """Quantize floating point scale into 32-bit int scale with a 6-bit shift.
+    This is to be used with 8-bit data.
+    """
+    mantissa, exponent = math.frexp(scale)
+    mantissa_scaled = mantissa * (1 << 31)
+    mantissa_scaled = int(util.round_away_zero(mantissa_scaled))
+    required_shift = 31 - exponent
+    assert 0 <= required_shift < (1 << 6)
+    return mantissa_scaled, required_shift
+
+
+def _reduced_quantize_scale(scale):
+    """A reduction of precision is required for 16 bit data."""
+    mantissa_scaled, required_shift = _quantize_scale(scale)
+    # This is max a signed 16-bit number could represent
+    max_reduced_mantissa_scaled = (1 << 15) - 1
+    # if the current value is larger than pre-scaled max_reduced_mantissa_scaled
+    # we need to saturate the anwser to max_reduced_mantissa_scaled
+    if mantissa_scaled >= max_reduced_mantissa_scaled << 16:
+        reduced_mantissa_scaled = max_reduced_mantissa_scaled
+    else:
+        reduced_mantissa_scaled = (mantissa_scaled + (1 << 15)) >> 16
+    reduced_shift = required_shift - 16
+    return reduced_mantissa_scaled, reduced_shift
+
+
+def _calculate_hw_bias_scales(
+    ifm_scale, weight_scales, ofm_scale, ifm_dtype, is_faf_tanh_sigmoid=False
+):
+    """This function will produce a scale that is calculated using scales of ifm,
+    weights and ofm. It is also important to note that if per-channel / per-value
+    quantization required they should go into hw bias scales"""
+    if is_faf_tanh_sigmoid:
+        ifm_scale = ifm_scale * 0x3000
+    if ifm_dtype == np.uint8:
+        bias_scales = [np.double(ifm_scale * ws) / np.double(ofm_scale) for ws in weight_scales]
+    else:
+        assert ifm_dtype in (np.int8, np.int16)
+        ifm_scale_dbl = np.double(ifm_scale)
+        ofm_scale_dbl = np.double(ofm_scale)
+        bias_scales = [ifm_scale_dbl * np.double(ws) / ofm_scale_dbl for ws in weight_scales]
+
+    if ifm_dtype == np.int16:
+        hw_bias_scales = [_reduced_quantize_scale(bs) for bs in bias_scales]
+    else:
+        assert ifm_dtype in (np.uint8, np.int8)
+        hw_bias_scales = [_quantize_scale(bs) for bs in bias_scales]
+
+    return hw_bias_scales
+
+
+def get_target_accel_type():
+    """This is a helper function to convert cli accelerator type str argument

Review comment:
       Clarified




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] NicolaLancellotti commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
NicolaLancellotti commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r696641076



##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,183 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out code generation.
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enum to obtain the correct index
+    of qnn.conv2d arguments.
+    """
+
+    IFM = 0
+    WEIGHTS = 1
+    IFM_ZERO_POINT = 2
+    WEIGHTS_ZERO_POINT = 3
+    IFM_SCALE = 4
+    WEIGHTS_SCALE = 5
+
+
+class RequantArgs(Enum):
+    """
+    This is a helper enum to obtain the correct index
+    of qnn.requantize arguments.
+    """
+
+    IFM_SCALE = 1
+    IFM_ZERO_POINT = 2
+    OFM_SCALE = 3
+    OFM_ZERO_POINT = 4
+
+
+class BiasAddArgs(Enum):
+    """
+    This is a helper enums to obtain the correct index
+    of qnn.bias_add arguments.
+    """
+
+    BIASES = 1
+
+
+class ClipArgs(Enum):
+    """
+    This is a helper enums to obtain the correct index
+    of clip arguments.
+    """
+
+    A_MIN = 1
+    A_MAX = 2
+
+
+def is_composite_func(func, name):
+    """
+    This method checks whether the call is to
+    a composite function of a given name.
+
+    Parameters
+    ----------
+    func : relay.Function
+        The header to be displayed along with the dump.
+
+    name : str
+        The candidate name to be checked
+
+    Returns
+    --------
+    a boolean
+    """
+
+    if not hasattr(func, "attrs"):
+        return False
+    if "Composite" not in func.attrs.keys():
+        return False
+    composite_name = func.attrs["Composite"]
+
+    if composite_name != name:
+        return False
+    return True
+
+
+def get_range_for_dtype_str(dtype):
+    """
+    Produce the min,max for a give data type.
+
+    Parameters
+    ----------
+    dtype : str
+        a type string (e.g., int8)
+
+    Returns
+    -------
+    type_info.min : int
+        the minimum of the range
+    type_info.max : int
+        the maximum of the range
+    """
+
+    try:
+        type_info = np.iinfo(dtype)
+    except ValueError:
+        type_info = np.finfo(dtype)
+    return type_info.min, type_info.max
+
+
+def round_away_zero(f):
+    """round the number away from zero towards +inf / -inf"""

Review comment:
       ```suggestion
       """Round the number away from zero towards +inf / -inf"""
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-905225146


   @jroesch Thanks for the comments! I've addressed majority of them bar TE parts.
   
   @mbaret Thanks for the clarification with the RFC and PR strategy.
   Would you be able take a look about comments related to TE parts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695435294



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r700175964



##########
File path: python/tvm/relay/backend/contrib/ethosu/te/convolution.py
##########
@@ -0,0 +1,199 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unused-argument
+"""Tensor Expressions for convolutions for the NPU"""
+from tvm import te
+from .dma import dma_ofm_compute, dma_ifm_compute
+
+
+def process_stride(stride):
+    """Process the striding into a common format.
+
+    Parameters
+    ----------
+    stride : Union[int, tuple, list]
+        The 2D striding.
+        int -> striding is the same in the height and width axis.
+        2D -> striding specified as (stride height, stride width).
+
+    Returns
+    -------
+    int
+        The stride in the height axis.
+    int
+        The stride in the width axis.
+
+    """
+    assert isinstance(stride, int) or len(stride) == 2
+    if isinstance(stride, int):
+        return stride, stride
+
+    return stride
+
+
+def process_dilation(dilation):
+    """Process the dilation into a common format.
+
+    Parameters
+    ----------
+    dilation : Union[int, tuple, list]
+        The 2D dilation.
+        int -> dilation is the same in the height and width axis.
+        2D -> dilation specified as (dilation height, dilation width).
+
+    Returns
+    -------
+    int
+        The dilation in the height axis.
+    int
+        The dilation in the width axis.
+
+    """
+    assert isinstance(dilation, int) or len(dilation) == 2
+    if isinstance(dilation, int):
+        return dilation, dilation
+
+    return dilation
+
+
+def conv2d_compute(
+    ifm,
+    weight,
+    scale_bias,
+    lut,
+    ifm_scale,
+    ifm_zero_point,
+    weight_zero_point,
+    ofm_scale,
+    ofm_zero_point,
+    strides,
+    padding,
+    dilation,
+    activation,
+    clip_min,
+    clip_max,
+    upscale,
+    ifm_layout,
+    ofm_layout,
+):
+    """A compute operator representing the capabilities of a 2D convolution for the NPU.
+
+    Parameters
+    ----------
+    ifm : te.Tensor
+        The Input Feature Map tensor (IFM).
+    weight : te.Tensor
+        The weight tensor.
+    scale_bias : te.Tensor
+        The packed per-channel weight scale and bias tensor.
+    lut : te.Tensor
+        The look-up table values to use if activation = "LUT".
+    ifm_scale : float
+        The quantization scale for the Input Feature Map tensor.
+    ifm_zero_point : int
+        The quantization zero point for the Input Feature Map tensor.
+    weight_zero_point : int
+        The quantization zero point for the weight tensor.
+    ofm_scale : float
+        The quantization scale for the Output Feature Map tensor.
+    ofm_zero_point : int
+        The quantization zero point for the Output Feature Map tensor.
+    strides : tuple
+        The 2 dimensional strides as (stride_height, stride_width).
+    padding : tuple
+        The 4 dimensional padding as (pad_top, pad_left, pad_bottom, pad_right).
+    dilation : Union[int, tuple, list]
+        The 2 dimensional dilation as (dilation_height, dilation_width).
+    activation : str
+        The activation function to use.
+            "NONE" - no activation function.
+            "CLIP" - clip the output between clip_min and clip_max.
+            "TANH" - tanh activation function.
+            "SIGMOID" - sigmoid activation function.
+            "LUT" - use a look-up table to perform the activation function.
+    clip_min : int
+        The minimum clipping value if activation = "CLIP".
+    clip_max : int
+        The maximum clipping value if activation = "CLIP".
+    upscale : str
+        The 2x2 upscaling mode to apply to the Input Feature Map tensor.
+            "NONE" - no upscaling.
+            "NEAREST" - upscale using nearest neighbour.
+            "ZEROS" - upscale using zeros.
+    ifm_layout : str
+        The layout of the Input Feature Map tensor. Can be "NHWC" or "NHCWB16".
+    ofm_layout : str
+        The layout of the Output Feature Map tensor. Can be "NHWC" or "NHCWB16".
+
+    Returns
+    -------
+    te.Tensor
+        The OFM tensor.
+
+    """
+    assert ifm.shape[0] == 1
+    assert ifm_layout in {"NHWC", "NHCWB16"}
+    assert ofm_layout in {"NHWC", "NHCWB16"}
+
+    stride_h, stride_w = strides
+    dilation_h, dilation_w = dilation
+    ofm_channels, kernel_h, kernel_w, ifm_channels = weight.shape
+
+    # Compute operation for the IFM DMA pipeline
+    dmaed_ifm = dma_ifm_compute(
+        ifm, ifm_layout, ifm_zero_point, ifm_scale, weight.shape[3], padding
+    )
+
+    # 2D Convolution compute operation
+    dilated_kernel_h = (kernel_h - 1) * dilation_h + 1
+    dilated_kernel_w = (kernel_w - 1) * dilation_w + 1

Review comment:
       I think we can address that when we introduce another operator that has similar logic. That way we can see if it's worth pulling out and if so what the correct interface should be.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] jroesch commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
jroesch commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-910739999


   @manupa-arm let's try to land this one tomorrow during (our) daytime? ideally we can get it green while Mark and I are sleeping and then land this and move on to part 3 and 4?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695439000



##########
File path: python/tvm/relay/op/contrib/ethosu.py
##########
@@ -0,0 +1,251 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Arm(R) Ethos(TM)-U NPU supported operators."""
+import numpy as np
+
+from tvm.relay.expr import Constant
+from tvm.relay.op.contrib.register import register_pattern_table
+from tvm.relay.dataflow_pattern import wildcard, is_op, is_constant
+from tvm.relay.backend.contrib.ethosu.util import QConv2DArgs
+from tvm.relay.backend.contrib.ethosu.util import BiasAddArgs
+from tvm.relay.backend.contrib.ethosu.util import RequantArgs
+from tvm.relay.backend.contrib.ethosu.util import get_dim_value
+from ethosu.vela import api as vapi
+
+
+def check_strides(strides):
+    """Checks whether strides are within the limits supported by the hardware"""
+    stride_range = (1, 3)
+    smin, smax = stride_range
+    if not smax >= strides[0] >= smin:
+        return False
+    if not smax >= strides[1] >= smin:
+        return False
+    return True
+
+
+def check_valid_dtypes(tensor_params):
+    """Check whether dtypes are supported by the hardware"""
+    supported_dtypes = (np.uint8, np.int8)
+    for tep in tensor_params:
+        # Check for dtypes
+        if np.dtype(tep.dtype) not in supported_dtypes:
+            return False
+        # Check for shape sizes
+        if any(dimlen > 65536 for dimlen in tep.shape):
+            return False
+    return True
+
+
+def check_weights(weights, dilation):
+    """Checks whether weight tensor is compatible with HW"""
+    dilated_height_range = (1, 64)
+    dilated_hxw_range = (1, 64 * 64)
+    weights_limit = 127 * 65536
+    dilated_width = (weights.shape[get_dim_value(weights.layout, "W")] - 1) * dilation[0] + 1
+    dilated_height = (weights.shape[get_dim_value(weights.layout, "H")] - 1) * dilation[1] + 1
+    dh_min, dh_max = dilated_height_range
+    if not dh_min <= dilated_height <= dh_max:
+        return False
+    dilated_hxw = dilated_height * dilated_width
+    dhxw_min, dhxw_max = dilated_hxw_range
+    if not dhxw_min <= dilated_hxw <= dhxw_max:
+        return False
+    # A saturation upper bound check for accumulators
+    weights.values = weights.values - weights.q_params.zero_point
+    axis = (
+        get_dim_value(weights.layout, "H"),
+        get_dim_value(weights.layout, "W"),
+        get_dim_value(weights.layout, "I"),
+    )
+    sum_weights = np.amax(np.sum(np.absolute(weights.values), axis=axis))
+    if not sum_weights <= weights_limit:
+        return False
+    return True
+
+
+def check_bias(bias):
+    """Check whether the bias values fit in 40 bits"""
+    if bias and bias.dtype == np.dtype("int64"):
+        valid = all(len(bin(bias_value)[2:]) <= 40 for bias_value in bias.values)
+        return valid
+    return True
+
+
+def check_batch_size(ifm):
+    """Checks for the number of batches vela currently supports"""
+    if ifm.shape[0] != 1:
+        return False
+    return True
+
+
+def check_dilation(dilation):
+    """Checks whether dilation is within the limits supported by the hardware"""
+    dilation_range = (1, 2)
+    dmin, dmax = dilation_range
+    if not dmin <= dilation[0] <= dmax:
+        return False
+    if not dmin <= dilation[1] <= dmax:
+        return False
+    return True
+
+
+def check_padding(padding, bounds):
+    """Checks whether padding is within the limits supported by the hardware"""
+    if len(padding) != 4 or len(bounds) != 4:
+        return False
+    top, left, bottom, right = padding
+    topb, leftb, bottomb, rightb = bounds
+    if top > topb or left > leftb or bottom > bottomb or right > rightb:
+        return False
+    return True
+
+
+class TensorParams:
+    """
+    This class will parse a tvm Expr along with quantization scale
+    and zero point to populate parameters that are required
+    for the creation of tensors in Vela.
+    """
+
+    def __init__(self, tensor, layout=None, scale=None, zero_point=None):
+        self.tensor = tensor
+        if isinstance(tensor, Constant):
+            self.values = tensor.data.asnumpy()
+        else:
+            self.values = None
+        self.dtype = tensor.checked_type.dtype
+        self.shape = [int(i) for i in tensor.checked_type.shape]
+        self.layout = layout
+
+        if scale is not None and zero_point is not None:
+            self.q_params = vapi.NpuQuantization(
+                scale.data.asnumpy().astype("float32"), zero_point.data.asnumpy().astype(self.dtype)
+            )
+        else:
+            # put default values
+            self.q_params = vapi.NpuQuantization(1.0, 0)
+
+
+class QnnConv2DParams:
+    """
+    This class will parse a Call to a ethosu.qnn_conv2d_clip composite function
+    and extract quantization information of all the associated tensors.
+    """
+
+    composite_name = "ethosu.qnn_conv2d"
+    # The hardware only supports padding upto the numbers as follows
+    padding_bounds = [31, 31, 32, 32]
+    activation_map = {"clip": "CLIP"}
+
+    def __init__(self, func_body):
+        activation = None
+        if str(func_body.op) in self.activation_map.keys():
+            activation = func_body
+            requantize_op = activation.args[0]
+        else:
+            requantize_op = func_body
+        bias_add = requantize_op.args[0]
+        qnn_conv2d = bias_add.args[0]
+        data_layout = qnn_conv2d.attrs.data_layout
+        kernel_layout = qnn_conv2d.attrs.kernel_layout
+        # We consider the weights & biases as params as it should be a Constant
+        self.weights = TensorParams(
+            qnn_conv2d.args[QConv2DArgs.weights.value],
+            kernel_layout,
+            qnn_conv2d.args[QConv2DArgs.weights_scale.value],
+            qnn_conv2d.args[QConv2DArgs.weights_zero_point.value],
+        )
+
+        self.biases = TensorParams(
+            bias_add.args[BiasAddArgs.biases.value],
+            data_layout,
+            requantize_op.args[RequantArgs.ifm_scale.value],
+            requantize_op.args[RequantArgs.ifm_zero_point.value],
+        )
+        self.ifm = TensorParams(
+            qnn_conv2d.args[QConv2DArgs.ifm.value],
+            data_layout,
+            qnn_conv2d.args[QConv2DArgs.ifm_scale.value],
+            qnn_conv2d.args[QConv2DArgs.ifm_zero_point.value],
+        )
+        self.ofm = TensorParams(
+            func_body,
+            data_layout,
+            requantize_op.args[RequantArgs.ofm_scale.value],
+            requantize_op.args[RequantArgs.ofm_zero_point.value],
+        )
+        self.padding = qnn_conv2d.attrs.padding
+        self.strides = qnn_conv2d.attrs.strides
+        self.dilation = qnn_conv2d.attrs.dilation
+        self.activation = activation
+
+        # If groups are equal to channel, its a depthwise_conv2d
+        self.groups = qnn_conv2d.attrs.groups
+        self.is_depthwise = False
+        channels_axis = {"HWIO": 3, "HWOI": 2}
+        if qnn_conv2d.attrs.groups == self.weights.shape[channels_axis[kernel_layout]]:
+            self.is_depthwise = True
+
+    def is_valid(self):
+        """
+        Checks whether QnnConv2D with Clip has compatible attributes with HW
+        """
+        tensor_params = [self.weights, self.ifm, self.ofm]
+        if not check_valid_dtypes(tensor_params):
+            return False
+        if not check_weights(self.weights, self.dilation):
+            return False
+        if not check_bias(self.biases):
+            return False
+        if not check_strides(self.strides):
+            return False
+        if not check_batch_size(self.ifm):
+            return False
+        if not check_dilation(self.dilation):
+            return False
+        if not check_padding(self.padding, self.padding_bounds):
+            return False
+        legal_groups = [1, self.ofm.shape[3]]
+        if self.groups not in legal_groups:
+            return False
+        # This should be a valid QnnDepthwise2DParams, not QnnConv2DParams
+        if self.is_depthwise:
+            return False
+        return True
+
+
+def qnn_conv2d_pattern():
+    """
+    Create pattern for qnn.conv2D with optional fused relu

Review comment:
       Done

##########
File path: src/relay/backend/contrib/ethosu/preprocess.cc
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include <tvm/ir/error.h>
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/attrs/annotation.h>
+#include <tvm/relay/expr.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/transform.h>
+
+#include <unordered_map>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "../../../op/make_op.h"
+
+namespace tvm {
+namespace relay {
+namespace contrib {
+namespace ethosu {
+
+/*!
+ * \brief This expression rewriter will traverse the graph to find calls
+ * to all external functions. If they have multiple inputs and/or
+ * multiple outputs, the following has to be done :
+ * 1) If multiple inputs are present, they needed to be concat before the call.
+ * 2) Inside the external function they need to be split again to their original inputs.
+ * 3) If there are multiple outputs, they need to be concat at the end of external function.
+ * 4) Then, the concat output again need to be split and made the original tuple output in the
+ * main.
+ */
+class ExternalFuncIOHandler : public ExprRewriter {
+ public:
+  explicit ExternalFuncIOHandler(IRModule& module) : module_(module) {}
+  int count = 0;
+
+  Function InferType(const Function& expr, const IRModule& m) {
+    IRModule mod(m);
+    mod->Update(mod->GetGlobalVar("main"), expr);
+    mod = transform::InferType()(mod);
+    return Downcast<Function>(mod->Lookup("main"));
+  }
+
+  /*!
+   * \brief This function will take shape and compute
+   * the scalar size value for it to be use to create
+   * flat single dimensional tensors.
+   */
+  int64_t CalcSize(const Array<Integer>& shape) {
+    int size = 1;
+    for (auto dim_sz : shape) {
+      size = size * Downcast<Integer>(dim_sz)->value;
+    }
+    return size;
+  }
+
+  /*!
+   * \brief This will take a tensor and create a flattened
+   * tensor to be used by the concat.
+   */
+  Expr CreateFlattenTensor(const Expr& input) {
+    auto ishape = Downcast<Array<Integer>>(Downcast<TensorType>(input->checked_type())->shape);
+    int flatten_size = CalcSize(ishape);
+    Array<Integer> oshape = {Integer(flatten_size)};
+    return MakeReshape(input, oshape);
+  }
+
+  /*!
+   * \brief This will take flattened tensors and create
+   * a single concat'd tensor.
+   */
+  Expr CreateConcatTensor(const Array<Expr>& inputs) {
+    auto tuple = Tuple(inputs);
+    return MakeConcatenate(tuple, 0);
+  }
+
+  /*!
+   * \brief This will take a flattened concat'd tensor and use the original inputs shapes
+   * to recreate a Tuple of the original set of tensors.
+   */
+  Expr CreateSplitReshapedTensors(const Expr& input, const Array<Expr>& original_args) {
+    Array<Array<Integer>> shapes;
+    Array<Integer> flatten_tensor_sizes;
+    Array<IndexExpr> split_indices;
+    Array<Expr> rets;
+
+    int total_size = 0;
+    for (auto orig_arg : original_args) {
+      auto shape = Downcast<Array<Integer>>(Downcast<TensorType>(orig_arg->checked_type())->shape);
+      shapes.push_back(shape);
+      flatten_tensor_sizes.push_back(CalcSize(shape));
+      if (total_size != 0) {
+        split_indices.push_back(total_size);
+      }
+      total_size += CalcSize(shape);
+    }
+    auto split_outs = MakeSplit(input, split_indices, 0);
+    for (unsigned int i = 0; i < shapes.size(); i++) {
+      auto split_out = TupleGetItem(split_outs, i);
+      split_out->checked_type_ = original_args[i]->checked_type_;
+      rets.push_back(MakeReshape(split_out, shapes[i]));
+    }
+    return Tuple(rets);
+  }
+
+  /*!
+   * \brief Modify the external function to split the input as the original compute
+   * as required originally. Moreover, the outputs will be flattened and concat'd
+   * to make a single output. Finaly, the external function should only have a single input
+   * and a single output.
+   */
+  Function ModifyExternalFunction(const Function& func, GlobalVar gv, const DataType& dtype) {

Review comment:
       Done

##########
File path: src/relay/backend/contrib/ethosu/preprocess.cc
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include <tvm/ir/error.h>
+#include <tvm/relay/analysis.h>
+#include <tvm/relay/attrs/annotation.h>
+#include <tvm/relay/expr.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/transform.h>
+
+#include <unordered_map>
+#include <unordered_set>
+#include <utility>
+#include <vector>
+
+#include "../../../op/make_op.h"
+
+namespace tvm {
+namespace relay {
+namespace contrib {
+namespace ethosu {
+
+/*!
+ * \brief This expression rewriter will traverse the graph to find calls
+ * to all external functions. If they have multiple inputs and/or
+ * multiple outputs, the following has to be done :
+ * 1) If multiple inputs are present, they needed to be concat before the call.
+ * 2) Inside the external function they need to be split again to their original inputs.
+ * 3) If there are multiple outputs, they need to be concat at the end of external function.
+ * 4) Then, the concat output again need to be split and made the original tuple output in the
+ * main.
+ */
+class ExternalFuncIOHandler : public ExprRewriter {
+ public:
+  explicit ExternalFuncIOHandler(IRModule& module) : module_(module) {}
+  int count = 0;
+
+  Function InferType(const Function& expr, const IRModule& m) {
+    IRModule mod(m);
+    mod->Update(mod->GetGlobalVar("main"), expr);
+    mod = transform::InferType()(mod);
+    return Downcast<Function>(mod->Lookup("main"));
+  }
+
+  /*!
+   * \brief This function will take shape and compute
+   * the scalar size value for it to be use to create
+   * flat single dimensional tensors.
+   */
+  int64_t CalcSize(const Array<Integer>& shape) {
+    int size = 1;
+    for (auto dim_sz : shape) {
+      size = size * Downcast<Integer>(dim_sz)->value;
+    }
+    return size;
+  }
+
+  /*!
+   * \brief This will take a tensor and create a flattened
+   * tensor to be used by the concat.
+   */
+  Expr CreateFlattenTensor(const Expr& input) {
+    auto ishape = Downcast<Array<Integer>>(Downcast<TensorType>(input->checked_type())->shape);
+    int flatten_size = CalcSize(ishape);
+    Array<Integer> oshape = {Integer(flatten_size)};
+    return MakeReshape(input, oshape);
+  }
+
+  /*!
+   * \brief This will take flattened tensors and create
+   * a single concat'd tensor.
+   */
+  Expr CreateConcatTensor(const Array<Expr>& inputs) {
+    auto tuple = Tuple(inputs);
+    return MakeConcatenate(tuple, 0);
+  }
+
+  /*!
+   * \brief This will take a flattened concat'd tensor and use the original inputs shapes
+   * to recreate a Tuple of the original set of tensors.
+   */
+  Expr CreateSplitReshapedTensors(const Expr& input, const Array<Expr>& original_args) {
+    Array<Array<Integer>> shapes;
+    Array<Integer> flatten_tensor_sizes;
+    Array<IndexExpr> split_indices;
+    Array<Expr> rets;
+
+    int total_size = 0;
+    for (auto orig_arg : original_args) {
+      auto shape = Downcast<Array<Integer>>(Downcast<TensorType>(orig_arg->checked_type())->shape);
+      shapes.push_back(shape);
+      flatten_tensor_sizes.push_back(CalcSize(shape));
+      if (total_size != 0) {
+        split_indices.push_back(total_size);
+      }
+      total_size += CalcSize(shape);
+    }
+    auto split_outs = MakeSplit(input, split_indices, 0);
+    for (unsigned int i = 0; i < shapes.size(); i++) {
+      auto split_out = TupleGetItem(split_outs, i);
+      split_out->checked_type_ = original_args[i]->checked_type_;
+      rets.push_back(MakeReshape(split_out, shapes[i]));
+    }
+    return Tuple(rets);
+  }
+
+  /*!
+   * \brief Modify the external function to split the input as the original compute
+   * as required originally. Moreover, the outputs will be flattened and concat'd
+   * to make a single output. Finaly, the external function should only have a single input
+   * and a single output.
+   */
+  Function ModifyExternalFunction(const Function& func, GlobalVar gv, const DataType& dtype) {
+    Array<Expr> inputs;
+    Var ifms;
+    if (func->params.size() > 1) {
+      Array<Array<Integer>> shapes;
+      Array<Integer> flatten_tensor_sizes;
+      Array<IndexExpr> split_indices;
+
+      auto func_name = gv->name_hint;
+      int total_size = 0;
+      for (auto input : func->params) {
+        auto shape = Downcast<Array<Integer>>(Downcast<TensorType>(input->checked_type())->shape);
+        shapes.push_back(shape);
+        auto flat_size = CalcSize(shape);
+        flatten_tensor_sizes.push_back(flat_size);
+        if (total_size != 0) {
+          split_indices.push_back(total_size);
+        }
+        total_size += flat_size;
+      }
+      Array<PrimExpr> ifms_shape = {total_size};
+      ifms = Var(func_name + "_ifms", TensorType(ifms_shape, dtype));
+      auto split_outs = MakeSplit(ifms, split_indices, 0);
+      for (unsigned int i = 0; i < shapes.size(); i++) {
+        auto split_out = TupleGetItem(split_outs, i);
+        split_out->checked_type_ = func->params[i]->checked_type();
+        inputs.push_back(MakeReshape(split_out, shapes[i]));
+      }
+    } else {
+      CHECK_EQ(func->params.size(), 1);
+      inputs.push_back(func->params[0]);
+      ifms = func->params[0];
+    }
+    Map<Var, Expr> bind_map;
+    CHECK_EQ(func->params.size(), inputs.size());
+    for (size_t i = 0; i < inputs.size(); i++) {
+      bind_map.Set(func->params[i], inputs[i]);
+    }
+    auto core_compute_expr = Bind(func->body, bind_map);
+
+    // Creation of wrapper inside the external function
+    Array<Var> params = {ifms};
+    if (func->body->IsInstance<TupleNode>()) {
+      auto tuple_out = func->body.as<TupleNode>();
+      Array<Expr> reshaped_outputs;
+      for (unsigned int i = 0; i < tuple_out->fields.size(); i++) {
+        auto out = Downcast<Tuple>(core_compute_expr)->fields[i];
+        out->checked_type_ = tuple_out->fields[i]->checked_type_;
+        reshaped_outputs.push_back(CreateFlattenTensor(out));
+      }
+      auto concat_out = CreateConcatTensor(reshaped_outputs);
+      auto f = Function(params, concat_out, concat_out->checked_type_, {}, func->attrs);
+      return InferType(f, this->module_);
+    } else {
+      auto f =
+          Function(params, core_compute_expr, core_compute_expr->checked_type_, {}, func->attrs);
+      return InferType(f, this->module_);
+    }
+  }
+
+  Expr Rewrite_(const CallNode* call, const Expr& post) final {
+    auto post_call = Downcast<Call>(post);
+
+    if (auto glb_var_node = post_call->op.as<GlobalVarNode>()) {
+      auto glb_var = GetRef<GlobalVar>(glb_var_node);
+      auto func = Downcast<Function>(module_->functions[glb_var]);
+
+      // If the number of inputs and output are 1 --> no need to do anything
+      if (post_call->args.size() == 1 && !func->body->IsInstance<TupleNode>()) {
+        return post;
+      }
+      if (auto compiler = func->GetAttr<String>(attr::kCompiler)) {
+        if (compiler == "ethosu") {
+          auto ext_input = std::move(post_call->args[0]);
+          auto arg_dtype = Downcast<TensorType>(post_call->args[0]->checked_type())->dtype;
+          if (post_call->args.size() > 1) {
+            Array<Expr> reshaped_inputs;
+            for (const auto& arg : post_call->args) {
+              // All arguments should be of same data type
+              CHECK_EQ(arg_dtype, Downcast<TensorType>(arg->checked_type())->dtype)
+                  << "Currently NPU external functions require all inputs to be of same data "
+                     "type";
+              reshaped_inputs.push_back(CreateFlattenTensor(arg));
+            }
+            ext_input = CreateConcatTensor(reshaped_inputs);
+          }
+          auto ext_func = ModifyExternalFunction(func, glb_var, arg_dtype);
+          Array<Expr> new_args = {ext_input};
+          module_->Add(glb_var, ext_func);
+          Expr new_call = Call(glb_var, new_args);
+          if (func->body->IsInstance<TupleNode>()) {
+            auto orginal_tuple_out = Downcast<Tuple>(func->body);
+            new_call = CreateSplitReshapedTensors(new_call, orginal_tuple_out->fields);
+          }
+          return std::move(new_call);
+        }
+      }
+    }
+    return post;
+  }
+
+ private:
+  IRModule module_;
+};
+
+IRModule PreprocessExternalFuncIO_(IRModule module) {

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695436408



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The relay expression for split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+class EthosUConv2DRewriter(DFPatternCallback):
+    """Convert conv2d related composite functions to ethosu_conv2d operators"""

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695437251



##########
File path: python/tvm/relay/backend/contrib/ethosu/preprocess.py
##########
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+"""Set of passes to pre-process the IRModule prior to codegen"""

Review comment:
       Add a bit more clarification. This is the place to hold relay IRModule preprocessing passes that are required.

##########
File path: python/tvm/relay/backend/contrib/ethosu/preprocess.py
##########
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+"""Set of passes to pre-process the IRModule prior to codegen"""

Review comment:
       Added a bit more clarification. This is the place to hold relay IRModule preprocessing passes that are required.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695437522



##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator

Review comment:
       Done

##########
File path: python/tvm/relay/backend/contrib/ethosu/util.py
##########
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Helper utility Enums and Functions used through out codegen
+
+The enums are there to indicate which argument of each relay operator
+corresponds with which input.
+e.g., input zero point of qnn.conv2d is 4th argument(3rd index)
+
+The rest of the utility functions are misc.
+Refer to the description inside such functions
+"""
+
+from enum import Enum
+import numpy as np
+
+from tvm import relay
+from tvm.relay.build_module import bind_params_by_name
+from tvm.relay.backend.contrib.ethosu import preprocess
+
+
+class QConv2DArgs(Enum):
+    """
+    This is a helper enums to access the correct index
+    qnn conv2d arguments
+    """
+
+    ifm = 0

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695439618



##########
File path: tests/python/contrib/test_ethosu/relay_ir_builder.py
##########
@@ -0,0 +1,295 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Helper module to build relay operations for testing"""

Review comment:
       This is a helper that uses the normal IRBuilder to create relay that get lowered from quantized models.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695436021



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.

Review comment:
       Done ( I think)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r700173929



##########
File path: python/tvm/relay/backend/contrib/ethosu/te/convolution.py
##########
@@ -0,0 +1,199 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unused-argument
+"""Tensor Expressions for convolutions for the NPU"""
+from tvm import te
+from .dma import dma_ofm_compute, dma_ifm_compute
+
+
+def process_stride(stride):
+    """Process the striding into a common format.
+
+    Parameters
+    ----------
+    stride : Union[int, tuple, list]
+        The 2D striding.
+        int -> striding is the same in the height and width axis.
+        2D -> striding specified as (stride height, stride width).
+
+    Returns
+    -------
+    int
+        The stride in the height axis.
+    int
+        The stride in the width axis.
+
+    """
+    assert isinstance(stride, int) or len(stride) == 2
+    if isinstance(stride, int):
+        return stride, stride
+
+    return stride
+
+
+def process_dilation(dilation):
+    """Process the dilation into a common format.
+
+    Parameters
+    ----------
+    dilation : Union[int, tuple, list]
+        The 2D dilation.
+        int -> dilation is the same in the height and width axis.
+        2D -> dilation specified as (dilation height, dilation width).
+
+    Returns
+    -------
+    int
+        The dilation in the height axis.
+    int
+        The dilation in the width axis.
+
+    """
+    assert isinstance(dilation, int) or len(dilation) == 2
+    if isinstance(dilation, int):
+        return dilation, dilation
+
+    return dilation

Review comment:
       You're right, let's get rid of them.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-910441110


   Apologies missed point 1. Some of the directory decisions were made to loosely align with the VTA integration (another TVM-powered ML accelerator). In that, special 'VTA versions' of certain operators are defined in the VTA directory which are tightly coupled to the VTA target and compilation pathway. We have much the same concept for Ethos-U - Tensor Expressions which make sense for our target and pretty much nothing else. I think because of this, putting them into TOPI could cause confusion as they're treated very differently from all the other TOPI ops. I also agree with @manupa-arm that the ease of navigation between these closely related components is desirable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r700190014



##########
File path: python/tvm/relay/backend/contrib/ethosu/te/dma.py
##########
@@ -0,0 +1,299 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unnecessary-lambda
+"""Tensor Expressions for operations supported by the DMA engine"""
+import tvm
+from tvm import te
+from tvm.topi.utils import equal_const_int
+
+
+def _pad_tensor(tensor, pad_before, pad_after=None):
+    """Generate a padded tensor.
+
+    Parameters
+    ----------
+    tensor : te.Tensor
+        The tensor to pad.
+    pad_before : tuple of int
+        The 'before' padding on each axis.
+    pad_after : tuple of int
+        The 'after' padding on each axis.
+    Returns
+    -------
+    _pad : callable
+        The padded tensor.
+
+    """
+    pad_after = pad_after or pad_before
+    dims = len(tensor.shape)
+    assert len(pad_before) == dims
+    assert len(pad_after) == dims
+
+    def _pad(*indices):
+        not_zero = []
+        index_tuple = []

Review comment:
       Done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695440184



##########
File path: tests/python/contrib/test_ethosu/test_legalize.py
##########
@@ -0,0 +1,345 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument
+
+import pytest
+
+pytest.importorskip("ethosu.vela")
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.backend.contrib import ethosu
+from tvm.relay.backend.contrib.ethosu import legalize, preprocess
+from tvm.relay.dataflow_pattern import *
+from tvm.relay.op.contrib.ethosu import *
+import relay_ir_builder
+
+
+def test_split_indices_legalize():
+    def create_graph(axis):
+        x = relay.var("x", shape=(1, 50, 50, 3))
+        x_relu = relay.nn.relu(x)
+        split_o = relay.split(x_relu, [5, 20, 45], axis).tuple_value
+        return relay.Function([x], split_o)
+
+    def expected_mod_axis1():
+        expected_ir_string = """
+        #[version = "0.0.5"]

Review comment:
       Thanks! at this point, if its not a necessary, we'd prefer to keep it like this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] ashutosh-arm edited a comment on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
ashutosh-arm edited a comment on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-914968687


   Yes, of course.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695436362



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The relay expression for split operator
+
+        Returns
+        -------
+        section_begins : list

Review comment:
       defined it in the description




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695434851



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm edited a comment on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm edited a comment on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-910615699


   I've addressed/anwsered existing comments, I ll take pass tommorow add type annotations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbaret commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbaret commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r700023373



##########
File path: python/tvm/relay/backend/contrib/ethosu/te/dma.py
##########
@@ -0,0 +1,299 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name,unnecessary-lambda
+"""Tensor Expressions for operations supported by the DMA engine"""

Review comment:
       Yeah you're pretty much spot on, the TE we create is very closely coupled to the functionality of our NPU and as such we don't want to rely on existing definitions which may be changed. We need it to remain identical so the TIR matching works and we attach some extra information to the TE (usually in the form of attributes) to ease that process.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] mbs-octoml commented on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-905090551


   Thanks for the extra context, esp the correct RFC :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695436188



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr

Review comment:
       Its the relay.Expr that is the call to split relay.Op




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695438323



##########
File path: python/tvm/relay/backend/contrib/ethosu/vela_api.py
##########
@@ -0,0 +1,314 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+conversions between TVM and Vela. Therefore, all interactions with the
+Vela API are supposed to go through this adapter, with the hope that
+any changes to Vela API, TVM only needs to change this file.
+The following conversion APIs are added :
+    *Obtaining the best block config
+    *Compressing weights
+    *Packing biases
+"""
+import logging
+import math
+import numpy as np
+from ethosu.vela import api as vapi
+
+from tvm.relay.backend.contrib.ethosu import util
+
+# pylint: disable=invalid-name
+logger = logging.getLogger("Ethos-U")
+
+VELA_TO_NP_DTYPES = {
+    vapi.NpuDataType.UINT8: np.uint8,
+    vapi.NpuDataType.UINT16: np.uint16,
+    vapi.NpuDataType.INT8: np.int8,
+    vapi.NpuDataType.INT16: np.int16,
+    vapi.NpuDataType.INT32: np.int32,
+}
+
+SCALE_BIAS_LENGTH = 10
+
+
+def get_optimal_block_config(npu_op, accel_type):
+    """
+    "The NPU's unit of work is known as a block. It will fetch block(s) from Input
+    Feature Map (IFM) and a compute block for Output Feature Map (OFM).
+    Therefore, we need to pick an optimal block configuration considering bandwidth
+    to bring IFM blocks and the number of OFM block computes need to happen
+    to cover the OFM as indicated by the npu op.
+
+    Parameters
+    ----------
+    npu_op : ethosu.vela.api.NpuOperation
+        The NPU operation and its params
+    accel_type : ethosu.vela.api.NpuAccelerator
+        The NPU accelerator variant
+    Returns
+    -------
+    ethosu.vela.api.NpuShape3d :
+        The optimal block config for the operator
+    """
+    all_valid_block_configs = vapi.npu_find_block_configs(npu_op, accel_type)
+    return _get_optimal_block_config(all_valid_block_configs)
+
+
+def _get_optimal_block_config(all_valid_block_configs):
+    """An internal function to get block config with largest depth
+    and then highest volume/area"""
+    assert isinstance(all_valid_block_configs, list)
+    for block_cfg in all_valid_block_configs:
+        assert isinstance(block_cfg, vapi.NpuShape3D)
+
+    # Getting the largest volume block for benchmarksing
+    all_valid_block_configs.sort(
+        key=lambda _cfg: _cfg.depth * _cfg.height * _cfg.width, reverse=True
+    )
+    largest_volume_block_config = all_valid_block_configs[0]
+    largest_volume = (
+        largest_volume_block_config.depth
+        * largest_volume_block_config.height
+        * largest_volume_block_config.width
+    )
+
+    all_valid_block_configs.sort(key=lambda _cfg: _cfg.depth, reverse=True)
+    max_d = all_valid_block_configs[0].depth
+    max_depth_block_configs = [_cfg for _cfg in all_valid_block_configs if _cfg.depth == max_d]
+    max_depth_block_configs.sort(key=lambda _cfg: _cfg.height * _cfg.width, reverse=True)
+    max_area = max_depth_block_configs[0].height * max_depth_block_configs[0].width
+    max_area_depth_block_configs = [
+        _cfg for _cfg in max_depth_block_configs if _cfg.height * _cfg.width == max_area
+    ]
+    # This to get a deterministic anwser everytime
+    max_area_depth_block_configs.sort(key=lambda _cfg: _cfg.height, reverse=True)
+    assert len(max_area_depth_block_configs) > 0
+    current_volume = (
+        max_area_depth_block_configs[0].depth
+        * max_area_depth_block_configs[0].height
+        * max_area_depth_block_configs[0].width
+    )
+    logger.info("Using block config=%s", max_area_depth_block_configs[0])
+    logger.info(
+        "Quality of the block config w.r.t. max volume block config=%s",
+        100.0 * (current_volume / largest_volume),
+    )
+    return max_area_depth_block_configs[0]
+
+
+def compress_weights(
+    weights,
+    weights_zp,
+    weights_layout,
+    ifm_bitdepth,
+    block_depth,
+    dilation,
+    accel_type,
+    is_depthwise=False,
+):
+    """Obtain compressed weights from vela
+
+    Parameters
+    ----------
+    weights : numpy.ndarray
+        The raw weights
+    weights_zp : int
+        The zero point of the weights
+    weights_layout : str
+        A string literal indicating the layout
+        Supported values : HWIO, HWOI, OHWI
+    ifm_bitdepth : int
+        The bit depth of the ifm the weights are used with
+    block_depth : int
+        The depth of the optimal block config for the operator
+    dilation : tuple
+        A tuple of 2 elements indicating dilation in h and w
+    accel_type : ethosu.vela.api.NpuAccelerator
+        The NPU accelerator variant
+    is_depthwise : bool, Optional
+        This indicates whether the weights are compressed for depthwise convolution
+
+    Returns
+    -------
+    compressed_weights : bytearray
+        Compressed weights
+    """
+    layout_transform_indices = {"HWIO": (3, 0, 1, 2), "HWOI": (2, 0, 1, 3), "OHWI": (0, 1, 2, 3)}
+    assert weights_layout in layout_transform_indices.keys()
+    assert isinstance(weights_zp, np.int64)
+    weights = weights.astype(np.int64) - weights_zp
+    # Vela needs the weights in OHWI layout
+    weights_ohwi = np.transpose(weights, layout_transform_indices[weights_layout])
+    shape_ohwi = [
+        weights.shape[layout_transform_indices[weights_layout][0]],
+        weights.shape[layout_transform_indices[weights_layout][1]],
+        weights.shape[layout_transform_indices[weights_layout][2]],
+        weights.shape[layout_transform_indices[weights_layout][3]],
+    ]
+    block_traversal = calculate_block_traversal_mode(is_depthwise, shape_ohwi, ifm_bitdepth)
+    compressed_weights = vapi.npu_encode_weights(
+        accelerator=accel_type,
+        weights_volume=weights_ohwi,
+        dilation_xy=dilation,
+        ifm_bitdepth=ifm_bitdepth,
+        ofm_block_depth=block_depth,
+        is_depthwise=is_depthwise,
+        block_traversal=block_traversal,
+    )
+    return compressed_weights
+
+
+def calculate_block_traversal_mode(is_depthwise, weights_shape_ohwi, ifm_bitdepth):
+    """Calculate a block traversal mode given whether the op is depthwise convolution,
+    shape of weights and bit-depth of the ifm.
+    """
+
+    if is_depthwise:
+        return vapi.NpuBlockTraversal.DEPTH_FIRST
+    # Determine which block traversal strategy has better DPU utilization
+    kernel_size = weights_shape_ohwi[1] * weights_shape_ohwi[2]
+    depth_utilization = weights_shape_ohwi[3] / util.round_up(
+        weights_shape_ohwi[3], 32 if ifm_bitdepth == 8 else 16
+    )
+    part_kernel_utilization = (weights_shape_ohwi[3] / util.round_up(weights_shape_ohwi[3], 8)) * (
+        kernel_size / util.round_up(kernel_size, 4 if ifm_bitdepth == 8 else 2)
+    )
+    if part_kernel_utilization >= depth_utilization or weights_shape_ohwi[3] <= 8:
+        # Part-kernel first is always better for ifm depths <= 8
+        return vapi.NpuBlockTraversal.PART_KERNEL_FIRST
+    return vapi.NpuBlockTraversal.DEPTH_FIRST
+
+
+def pack_biases(
+    biases,
+    ifm_scale,
+    ifm_dtype,
+    weight_scales,
+    ofm_scale,
+    is_activation_tanh_or_sigmoid=False,
+):
+    """
+    Obtain packed bias bytearray as the hardware requires from

Review comment:
       explained further




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695436801



##########
File path: python/tvm/relay/backend/contrib/ethosu/legalize.py
##########
@@ -0,0 +1,200 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+""" A set of passes to legalize some of operations for the NPU"""
+import numpy as np
+
+import tvm
+from tvm import relay
+from tvm.relay.dataflow_pattern import DFPatternCallback
+from tvm.relay.dataflow_pattern import wildcard
+from tvm.relay.dataflow_pattern import is_op
+from tvm.relay.dataflow_pattern import rewrite
+from tvm.relay.backend.contrib.ethosu import op as ethosu_ops
+from tvm.relay.backend.contrib.ethosu.errors import UnsupportedLayout
+from tvm.relay.backend.contrib.ethosu import vela_api
+from tvm.relay.op.contrib import ethosu as ethosu_patterns
+
+
+class SplitRewriter(DFPatternCallback):
+    """Convert split operations to bunch of strided_slice operations,
+    because codegen is going to be based on strided_slices that are
+    close to in/out boxes of Vela High-Level Command Stream (HLCS).
+    Moreover, Vela HLCS is a high-level description of the supported
+    hardware operator.
+    """
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.split_in = wildcard()
+        self.pattern = is_op("split")(self.split_in)
+
+    @staticmethod
+    def get_section_begin_coords(split):
+        """Currently, the split can take an array of indices or an integer
+        indicating the number of splits. This helper functions unifies
+        this by making it a array of section begins.
+
+        Parameters
+        ----------
+        split : relay.Expr
+            The relay expression for split operator
+
+        Returns
+        -------
+        section_begins : list
+            A list containing integers corresponding to section
+            begins
+        """
+        indices_or_sections = split.attrs.indices_or_sections
+        input_shape = split.args[0].checked_type.shape
+        split_axis = split.attrs.axis
+
+        if isinstance(indices_or_sections, tvm.ir.container.Array):
+            # 0 is the beginning of the first section.
+            return [0] + list(indices_or_sections)
+        split_axis_len = input_shape[split_axis].value
+        section_length = split_axis_len // indices_or_sections.value
+        section_begins = list(range(0, split_axis_len, section_length))
+        return section_begins
+
+    def callback(self, pre, post, node_map):
+        splits_types = dict()
+        split_input = post.args[0]
+        for idx, field_type in enumerate(post.checked_type.fields):
+            split = relay.TupleGetItem(post, idx)
+            splits_types[split] = field_type
+
+        split_begins = list()
+        split_ends = list()
+        section_begins_in_split_axis = self.get_section_begin_coords(post)
+        for split_cord in section_begins_in_split_axis:
+            # first begin is [0, 0, ... , 0]
+            begin_shape = [0 for i in range(len(split_input.checked_type.shape))]
+            begin_shape[post.attrs.axis] = split_cord
+            split_begins.append(begin_shape)
+
+            end_shape = list(split_input.checked_type.shape)
+            # Only the split axis coordinate changes
+            end_shape[post.attrs.axis] = split_cord
+            split_ends.append(end_shape)
+
+        # Coordinates needs to be shifted left because beginning
+        # of the next section is the end of the previous
+        split_ends = split_ends[1:]
+        # Last section end is the shape of the tensor itself.
+        split_ends.append(list(split_input.checked_type.shape))
+
+        strided_slices = list()
+        for sb, se in zip(split_begins, split_ends):
+            strided_slices.append(relay.strided_slice(split_input, sb, se))
+
+        return relay.Tuple(strided_slices)
+
+
+class EthosUConv2DRewriter(DFPatternCallback):
+    """Convert conv2d related composite functions to ethosu_conv2d operators"""
+
+    def __init__(self):
+        super().__init__(require_type=True)
+        self.pattern = (wildcard().has_attr({"Composite": "ethosu.qnn_conv2d"}))(wildcard())
+
+    def callback(self, pre, post, node_map):
+        params = ethosu_patterns.QnnConv2DParams(post.op.body)
+        params.ifm.tensor = post.args[0]
+        channels_map = {
+            "NHWC": 3,
+        }
+        if str(params.ofm.layout) not in channels_map.keys():
+            raise UnsupportedLayout(str(params.ofm.layout))
+        kernel_size_map = {
+            "HWIO": params.weights.shape[0:2],
+            "OHWI": params.weights.shape[1:3],
+            "HWOI": params.weights.shape[0:2],
+        }
+        if str(params.weights.layout) not in kernel_size_map.keys():
+            raise UnsupportedLayout(str(params.weights.layout))
+        activation_map = {"clip": "CLIP"}
+        weight_to_ohwi_transform_map = {"HWIO": [3, 0, 1, 2]}
+        weights_values = params.weights.values
+        weights_values_ohwi = np.transpose(
+            weights_values, weight_to_ohwi_transform_map[str(params.weights.layout)]
+        )
+        if params.activation:
+            activation = activation_map[params.activation.op.name]
+            clip_min = int(params.activation.attrs.a_min)
+            clip_max = int(params.activation.attrs.a_max)
+        else:
+            activation = "NONE"
+            clip_min = 0
+            clip_max = 0
+        scale_bias = vela_api.pack_biases(
+            biases=params.biases.tensor.data.asnumpy(),
+            ifm_scale=params.ifm.q_params.scale_f32,
+            ifm_dtype=np.dtype(params.ifm.dtype),
+            weight_scales=params.weights.q_params.scale_f32,
+            ofm_scale=params.ofm.q_params.scale_f32,
+            is_activation_tanh_or_sigmoid=activation in ["TANH", "SIGMOID"],
+        )
+        ethosu_conv2d = ethosu_ops.ethosu_conv2d(
+            ifm=post.args[0],
+            weight=relay.const(weights_values_ohwi, params.weights.values.dtype),
+            scale_bias=relay.const(scale_bias, "uint8"),
+            lut=relay.const([], dtype="int8"),
+            ifm_scale=float(params.ifm.q_params.scale_f32),
+            ifm_zero_point=int(params.ifm.q_params.zero_point),
+            weight_zero_point=int(params.weights.q_params.zero_point),
+            ofm_scale=float(params.ofm.q_params.scale_f32),
+            ofm_zero_point=int(params.ofm.q_params.zero_point),
+            kernel_shape=kernel_size_map[str(params.weights.layout)],
+            ofm_channels=params.ofm.shape[channels_map[str(params.ofm.layout)]],
+            strides=params.strides,
+            padding=params.padding,
+            dilation=params.dilation,
+            activation=activation,
+            clip_min=clip_min,
+            clip_max=clip_max,
+            upscale="NONE",
+            ifm_layout=str(params.ifm.layout),
+            ofm_layout=str(params.ofm.layout),
+        )
+        return ethosu_conv2d
+
+
+class LegalizeEthosU:
+    """This is the wrapper class to call graph-rewrites to perform graph transformation

Review comment:
       Converted to a pass

##########
File path: python/tvm/relay/backend/contrib/ethosu/preprocess.py
##########
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+"""Set of passes to pre-process the IRModule prior to codegen"""
+from . import _ffi_api
+
+
+def preprocess_ext_io():

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm commented on a change in pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on a change in pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#discussion_r695434342



##########
File path: python/tvm/relay/backend/contrib/__init__.py
##########
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""external backend codegen modules for relay."""

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm edited a comment on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm edited a comment on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-905225146


   @jroesch Thanks for the comments! I've addressed majority of them bar TE parts.
   
   @mbaret Thanks for the clarification with the RFC and PR strategy.
   Would you be able take a look on the comments related to TE parts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] manupa-arm edited a comment on pull request #8795: [2/6] Arm(R) Ethos(TM)-U NPU Relay passes and Conv2D op

Posted by GitBox <gi...@apache.org>.
manupa-arm edited a comment on pull request #8795:
URL: https://github.com/apache/tvm/pull/8795#issuecomment-905225146


   @jroesch Thanks for the comments! I've addressed majority of them bar TE parts.
   
   Also regarding the python cpp split, the reason is some of the passes need to access Vela APIs that are in python. Therefore, we have decided to keep such passes as well as the integration in python.
   
   @mbaret Thanks for the clarification with the RFC and PR strategy.
   Would you be able take a look on the comments related to TE parts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org