You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/09/30 08:27:22 UTC
[GitHub] [tvm] ashutosh-arm opened a new pull request, #12950: [CMSIS-NN] Support for int16 conv2d
ashutosh-arm opened a new pull request, #12950:
URL: https://github.com/apache/tvm/pull/12950
Support for int16 Conv2d via CMSIS-NN
-Pattern matching and RelayToTIR introduce int16 support
-Added new context buffer size APIs for int16 Conv2d
-Added int16 variants to integration and buffer size tests
cc @lhutton1 @Mousius @leandron @grant-arm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] ashutosh-arm commented on a diff in pull request #12950: [CMSIS-NN] Support for int16 conv2d
Posted by GitBox <gi...@apache.org>.
ashutosh-arm commented on code in PR #12950:
URL: https://github.com/apache/tvm/pull/12950#discussion_r984865257
##########
python/tvm/relay/op/contrib/cmsisnn.py:
##########
@@ -145,12 +143,29 @@ def check_qnn_conv2d(pattern):
):
is_depthwise = True
+ # check if dtypes are supported for the following entities
+ # (input_dtype, weight_dtype, bias_dtype, out_dtype, pattern_dtype)
+ are_dtypes_valid = False
+ if bias_add:
+ bias_dtype = bias_add.args[1].checked_type.dtype
+ else:
+ bias_dtype = "int32" if conv2d_input.checked_type.dtype == "int8" else "int64"
+ valid_dtypes = None
+ if conv2d_input.checked_type.dtype == "int8":
+ valid_dtypes = ("int8", "int8", "int32", "int32", "int8")
+ elif conv2d_input.checked_type.dtype == "int16":
+ valid_dtypes = ("int16", "int8", "int64", "int64", "int16")
+ if (
+ conv2d_input.checked_type.dtype,
Review Comment:
makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] ashutosh-arm commented on a diff in pull request #12950: [CMSIS-NN] Support for int16 conv2d
Posted by GitBox <gi...@apache.org>.
ashutosh-arm commented on code in PR #12950:
URL: https://github.com/apache/tvm/pull/12950#discussion_r984863090
##########
src/relay/backend/contrib/cmsisnn/relay_to_tir.cc:
##########
@@ -133,6 +133,22 @@ class RelayToTIRVisitor : public MixedModeMutator {
} else {
conv2d_call = requantize_input;
}
+ int32_t dtype_bits = conv2d_call->args[0]->type_as<TensorTypeNode>()->dtype.bits();
+
+ // Determine bitwidth of buffers based on input dtype
+ int32_t input_bits = 8;
+ int32_t filter_bits = 8;
+ int32_t bias_bits = 32;
+ int32_t output_bits = 8;
+ int32_t context_buffer_bits = 8;
+ bool is_int16 = false;
+ if (dtype_bits == 16) {
+ is_int16 = true;
+ input_bits = 16;
+ bias_bits = 64;
+ output_bits = 16;
+ context_buffer_bits = 16;
+ }
Review Comment:
Function is unreachable for other types because of two reasons: pattern matching filters them out and the function is only called when a specific function name is seen on the composite function. Still think a check is not around dtypes? Let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] ashutosh-arm commented on a diff in pull request #12950: [CMSIS-NN] Support for int16 conv2d
Posted by GitBox <gi...@apache.org>.
ashutosh-arm commented on code in PR #12950:
URL: https://github.com/apache/tvm/pull/12950#discussion_r984860150
##########
tests/python/contrib/test_cmsisnn/test_conv2d.py:
##########
@@ -249,23 +256,25 @@ def test_conv2d_symmetric_padding_int8(
kernel_size = (3, 3)
strides = (1, 1)
dilation = (1, 1)
- dtype = "int8"
groups = 1
- weight_format = "HWIO"
+ input_zero_point = input_zero_point if dtype == "int8" else 0
Review Comment:
input_zero_point is not supported by TFLM when dtype == int16. I've added a comment about it. Thanks for pointing out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] lhutton1 commented on a diff in pull request #12950: [CMSIS-NN] Support for int16 conv2d
Posted by GitBox <gi...@apache.org>.
lhutton1 commented on code in PR #12950:
URL: https://github.com/apache/tvm/pull/12950#discussion_r985592946
##########
src/relay/backend/contrib/cmsisnn/relay_to_tir.cc:
##########
@@ -133,6 +133,22 @@ class RelayToTIRVisitor : public MixedModeMutator {
} else {
conv2d_call = requantize_input;
}
+ int32_t dtype_bits = conv2d_call->args[0]->type_as<TensorTypeNode>()->dtype.bits();
+
+ // Determine bitwidth of buffers based on input dtype
+ int32_t input_bits = 8;
+ int32_t filter_bits = 8;
+ int32_t bias_bits = 32;
+ int32_t output_bits = 8;
+ int32_t context_buffer_bits = 8;
+ bool is_int16 = false;
+ if (dtype_bits == 16) {
+ is_int16 = true;
+ input_bits = 16;
+ bias_bits = 64;
+ output_bits = 16;
+ context_buffer_bits = 16;
+ }
Review Comment:
Happy to leave :)
##########
tests/python/contrib/test_cmsisnn/test_conv2d.py:
##########
@@ -249,23 +256,25 @@ def test_conv2d_symmetric_padding_int8(
kernel_size = (3, 3)
strides = (1, 1)
dilation = (1, 1)
- dtype = "int8"
groups = 1
- weight_format = "HWIO"
+ input_zero_point = input_zero_point if dtype == "int8" else 0
Review Comment:
Ah I see thanks! Seems like there should be a check in the partitioning for this case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] lhutton1 commented on a diff in pull request #12950: [CMSIS-NN] Support for int16 conv2d
Posted by GitBox <gi...@apache.org>.
lhutton1 commented on code in PR #12950:
URL: https://github.com/apache/tvm/pull/12950#discussion_r984410551
##########
src/relay/backend/contrib/cmsisnn/relay_to_tir.cc:
##########
@@ -133,6 +133,22 @@ class RelayToTIRVisitor : public MixedModeMutator {
} else {
conv2d_call = requantize_input;
}
+ int32_t dtype_bits = conv2d_call->args[0]->type_as<TensorTypeNode>()->dtype.bits();
+
+ // Determine bitwidth of buffers based on input dtype
+ int32_t input_bits = 8;
+ int32_t filter_bits = 8;
+ int32_t bias_bits = 32;
+ int32_t output_bits = 8;
+ int32_t context_buffer_bits = 8;
+ bool is_int16 = false;
+ if (dtype_bits == 16) {
+ is_int16 = true;
+ input_bits = 16;
+ bias_bits = 64;
+ output_bits = 16;
+ context_buffer_bits = 16;
+ }
Review Comment:
Might be a good idea to produce an error if say `dtype_bits=32`?
##########
python/tvm/relay/op/contrib/cmsisnn.py:
##########
@@ -145,12 +143,29 @@ def check_qnn_conv2d(pattern):
):
is_depthwise = True
+ # check if dtypes are supported for the following entities
+ # (input_dtype, weight_dtype, bias_dtype, out_dtype, pattern_dtype)
+ are_dtypes_valid = False
+ if bias_add:
+ bias_dtype = bias_add.args[1].checked_type.dtype
+ else:
+ bias_dtype = "int32" if conv2d_input.checked_type.dtype == "int8" else "int64"
+ valid_dtypes = None
+ if conv2d_input.checked_type.dtype == "int8":
+ valid_dtypes = ("int8", "int8", "int32", "int32", "int8")
+ elif conv2d_input.checked_type.dtype == "int16":
+ valid_dtypes = ("int16", "int8", "int64", "int64", "int16")
+ if (
+ conv2d_input.checked_type.dtype,
Review Comment:
nit: since its used a few times, probably worth making it a variable?
##########
tests/python/contrib/test_cmsisnn/test_conv2d.py:
##########
@@ -249,23 +256,25 @@ def test_conv2d_symmetric_padding_int8(
kernel_size = (3, 3)
strides = (1, 1)
dilation = (1, 1)
- dtype = "int8"
groups = 1
- weight_format = "HWIO"
+ input_zero_point = input_zero_point if dtype == "int8" else 0
Review Comment:
Curious, why is this needed?
##########
tests/python/contrib/test_cmsisnn/test_conv2d.py:
##########
@@ -915,13 +943,15 @@ def test_relay_conv2d_cmsisnn_depthwise_int8(
def parameterize_for_invalid_model(test):
"""Generates non int8 inputs"""
Review Comment:
nit: update docstring
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] ashutosh-arm commented on a diff in pull request #12950: [CMSIS-NN] Support for int16 conv2d
Posted by GitBox <gi...@apache.org>.
ashutosh-arm commented on code in PR #12950:
URL: https://github.com/apache/tvm/pull/12950#discussion_r984858391
##########
tests/python/contrib/test_cmsisnn/test_conv2d.py:
##########
@@ -915,13 +943,15 @@ def test_relay_conv2d_cmsisnn_depthwise_int8(
def parameterize_for_invalid_model(test):
"""Generates non int8 inputs"""
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org