You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/11/06 10:11:55 UTC
[GitHub] [incubator-tvm] lsy643 opened a new pull request #6864: Qnn for first conv
lsy643 opened a new pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864
The Current `Partition` Pass for quantize will not insert `annotation.cast_hint` and `annotation.stop_fusion` for the first `Conv2D` of the model.
By adding there two lines in `def conv2d_partition_function`, the first `Conv2D` will be quantized as expected
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] lsy643 commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
lsy643 commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r520265346
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
@giuseros
Yes, without setting `partition_conversions=fully_integral` will make this test case meaningless.
But the whole thing is, I want to make the first `Conv` of a `Resnet` can also be quantized. Therefore, I add those two lines into `conv2d_partition_function`. However, with the two lines, the test case `test_conv2d_partition` and `test_multiple_arg_conversions_partition` can not pass.
I have also tried to use `relay.quantize.qconfig(calibrate_mode="global_scale", global_scale=8.0, partition_conversions="enabled")` when a model is quantized, but the program will just crash.
So, would you like to give some suggestion to make the first `Conv` of a `Resnet` can also be quantized without modifying the test cases?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] lsy643 commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
lsy643 commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r521009413
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
@giuseros
By the way, after I synchronized TVM repo last night, just adding those two lines would not make the first `Conv` get quantized neither. The partition pass works as expected. I am digging into it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] lsy643 commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
lsy643 commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r521009413
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
@giuseros
I have figured it out. It has not so much about the annotation pass.
Use `with relay.quantize.qconfig(calibrate_mode="global_scale", global_scale=8.0, skip_conv_layers=[])` can make the first conv works fine.
I am just curious why the default value for `skip_conv_layers` is `[0]`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] lsy643 commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
lsy643 commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r520993062
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
@giuseros
Yes, you are right.
For example, the input relay module could be:
```
fn (%input0: Tensor[(1, 3, 224, 224), float32], %conv1.weight: Tensor[(64, 3, 7, 7), float32], %conv2.weight: Tensor[(128, 64, 3, 3), float32]) {
%0 = nn.conv2d(%input0, %conv1.weight, strides=[2, 2], padding=[3, 3, 3, 3], channels=64, kernel_size=[7, 7]);
%1 = nn.relu(%0);
%2 = nn.max_pool2d(%1, pool_size=[3, 3], strides=[2, 2], padding=[1, 1, 1, 1]);
%3 = nn.conv2d(%2, %conv2.weight, padding=[1, 1, 1, 1], channels=128, kernel_size=[3, 3]);
nn.relu(%3)
}
```
And the mod after quantize is:
```
fn (%input0: Tensor[(1, 3, 224, 224), float32]) -> Tensor[(1, 128, 56, 56), float32] {
%0 = nn.conv2d(%input0, meta[relay.Constant][0] /* ty=Tensor[(64, 3, 7, 7), float32] */, strides=[2, 2], padding=[3, 3, 3, 3], channels=64, kernel_size=[7, 7]) /* ty=Tensor[(1, 64, 112, 112), float32] */;
%1 = nn.relu(%0) /* ty=Tensor[(1, 64, 112, 112), float32] */;
%2 = nn.max_pool2d(%1, pool_size=[3, 3], strides=[2, 2], padding=[1, 1, 1, 1]) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%3 = annotation.stop_fusion(%2) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%4 = multiply(%3, 16f /* ty=float32 */) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%5 = round(%4) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%6 = clip(%5, a_min=-127f, a_max=127f) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%7 = cast(%6, dtype="int8") /* ty=Tensor[(1, 64, 56, 56), int8] */;
%8 = nn.conv2d(%7, meta[relay.Constant][1] /* ty=Tensor[(128, 64, 3, 3), int8] */, padding=[1, 1, 1, 1], channels=128, kernel_size=[3, 3], out_dtype="int32") /* ty=Tensor[(1, 128, 56, 56), int32] */;
%9 = nn.relu(%8) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%10 = add(%9, 256 /* ty=int32 */) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%11 = right_shift(%10, 9 /* ty=int32 */) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%12 = clip(%11, a_min=-127f, a_max=127f) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%13 = cast(%12, dtype="int8") /* ty=Tensor[(1, 128, 56, 56), int8] */;
%14 = annotation.stop_fusion(%13) /* ty=Tensor[(1, 128, 56, 56), int8] */;
%15 = cast(%14, dtype="float32") /* ty=Tensor[(1, 128, 56, 56), float32] */;
multiply(%15, 0.0625f /* ty=float32 */) /* ty=Tensor[(1, 128, 56, 56), float32] */
}
```
Clearly, you can see that the two `nn.conv2d` have different input type, weight type and output type
I can provide a script to reproduce this issue, and you may need to install Pytorch and torchvision by `pip install torch==1.7.0` and `pip install torchvision==0.8.1`
```
import tvm
from tvm import relay
import torch
import torch.nn as nn
import torchvision
import numpy as np
class ConvSimple(nn.Module):
def __init__(self):
super(ConvSimple, self).__init__()
self.inplanes = 64
self.dilation = 1
norm_layer = nn.BatchNorm2d
self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=7, stride=2, padding=3, bias=False)
self.relu = nn.ReLU(inplace=True)
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
self.conv2 = nn.Conv2d(self.inplanes, 128, kernel_size=3, stride=1, padding=1,bias=False)
self.relu2 = nn.ReLU(inplace=True)
self.conv3 = nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1, bias=False)
self.relu3 = nn.ReLU(inplace=True)
self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
for m in self.modules():
if isinstance(m, nn.Conv2d):
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
def forward(self, x):
x = self.conv1(x)
x = self.relu(x)
x = self.maxpool(x)
x = self.conv2(x)
x = self.relu2(x)
return x
def get_conv_simple_script_module():
conv_simple = ConvSimple()
conv_simple.eval()
input_shape = [1, 3, 224, 224]
input_data = torch.randn(input_shape)
traced_conv_simple = torch.jit.trace(conv_simple, input_data).eval()
return traced_conv_simple
def main():
conv_simple_script_module = get_conv_simple_script_module()
img = np.random.normal(size=[1, 3, 224, 224])
input_name = "input0"
shape_list = [(input_name, img.shape)]
mod, params = relay.frontend.from_pytorch(conv_simple_script_module, shape_list)
print('\n\n**** original mod: \n', mod['main'])
with relay.quantize.qconfig(calibrate_mode="global_scale", global_scale=8.0):
qnn1_mod = relay.quantize.quantize(mod, params)
print('\n\n**** quantized mod: ', qnn1_mod['main'])
with relay.quantize.qconfig(calibrate_mode="global_scale", global_scale=8.0, partition_conversions="enabled"):
qnn2_mod = relay.quantize.quantize(mod, params)
print('quantize_inputs: ', qnn2_mod['quantize_inputs'])
print('quantized_main: ', qnn2_mod['quantized_main'])
print('dequantize_outputs: ', qnn2_mod['dequantize_outputs'])
if __name__ == '__main__':
main()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r521715708
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
Just read this comment. You already figure out. I am not sure about why. But, my guess would be that empirically the folks working on quantization might have seen big accuracy loss.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] lsy643 commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
lsy643 commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r521867350
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
@anijain2305 @giuseros
Thanks a lot. Since this PR is no longer need, I will close it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] lsy643 commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
lsy643 commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r520993062
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
@giuseros
Yes, you are right.
For example, the input relay module could be:
```
fn (%input0: Tensor[(1, 3, 224, 224), float32], %conv1.weight: Tensor[(64, 3, 7, 7), float32], %conv2.weight: Tensor[(128, 64, 3, 3), float32]) {
%0 = nn.conv2d(%input0, %conv1.weight, strides=[2, 2], padding=[3, 3, 3, 3], channels=64, kernel_size=[7, 7]);
%1 = nn.relu(%0);
%2 = nn.max_pool2d(%1, pool_size=[3, 3], strides=[2, 2], padding=[1, 1, 1, 1]);
%3 = nn.conv2d(%2, %conv2.weight, padding=[1, 1, 1, 1], channels=128, kernel_size=[3, 3]);
nn.relu(%3)
}
```
And the mod after quantize is:
```
fn (%input0: Tensor[(1, 3, 224, 224), float32]) -> Tensor[(1, 128, 56, 56), float32] {
%0 = nn.conv2d(%input0, meta[relay.Constant][0] /* ty=Tensor[(64, 3, 7, 7), float32] */, strides=[2, 2], padding=[3, 3, 3, 3], channels=64, kernel_size=[7, 7]) /* ty=Tensor[(1, 64, 112, 112), float32] */;
%1 = nn.relu(%0) /* ty=Tensor[(1, 64, 112, 112), float32] */;
%2 = nn.max_pool2d(%1, pool_size=[3, 3], strides=[2, 2], padding=[1, 1, 1, 1]) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%3 = annotation.stop_fusion(%2) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%4 = multiply(%3, 16f /* ty=float32 */) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%5 = round(%4) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%6 = clip(%5, a_min=-127f, a_max=127f) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%7 = cast(%6, dtype="int8") /* ty=Tensor[(1, 64, 56, 56), int8] */;
%8 = nn.conv2d(%7, meta[relay.Constant][1] /* ty=Tensor[(128, 64, 3, 3), int8] */, padding=[1, 1, 1, 1], channels=128, kernel_size=[3, 3], out_dtype="int32") /* ty=Tensor[(1, 128, 56, 56), int32] */;
%9 = nn.relu(%8) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%10 = add(%9, 256 /* ty=int32 */) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%11 = right_shift(%10, 9 /* ty=int32 */) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%12 = clip(%11, a_min=-127f, a_max=127f) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%13 = cast(%12, dtype="int8") /* ty=Tensor[(1, 128, 56, 56), int8] */;
%14 = annotation.stop_fusion(%13) /* ty=Tensor[(1, 128, 56, 56), int8] */;
%15 = cast(%14, dtype="float32") /* ty=Tensor[(1, 128, 56, 56), float32] */;
multiply(%15, 0.0625f /* ty=float32 */) /* ty=Tensor[(1, 128, 56, 56), float32] */
}
```
Clearly, you can see that the two `nn.conv2d` have different input type, weight type and output type
I can provide a script to reproduce this issue, and you may need to install Pytorch and torchvision by `pip install torch==1.7.0` and `pip install torchvision==0.8.1`
```
import tvm
from tvm import relay
import torch
import torch.nn as nn
import torchvision
import numpy as np
class ConvSimple(nn.Module):
def __init__(self):
super(ConvSimple, self).__init__()
self.inplanes = 64
self.dilation = 1
norm_layer = nn.BatchNorm2d
self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=7, stride=2, padding=3, bias=False)
self.relu = nn.ReLU(inplace=True)
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
self.conv2 = nn.Conv2d(self.inplanes, 128, kernel_size=3, stride=1, padding=1,bias=False)
self.relu2 = nn.ReLU(inplace=True)
self.conv3 = nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1, bias=False)
self.relu3 = nn.ReLU(inplace=True)
self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
for m in self.modules():
if isinstance(m, nn.Conv2d):
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
def forward(self, x):
x = self.conv1(x)
x = self.relu(x)
x = self.maxpool(x)
x = self.conv2(x)
x = self.relu2(x)
return x
def get_conv_simple_script_module():
conv_simple = ConvSimple()
conv_simple.eval()
input_shape = [1, 3, 224, 224]
input_data = torch.randn(input_shape)
traced_conv_simple = torch.jit.trace(conv_simple, input_data).eval()
return traced_conv_simple
def main():
conv_simple_script_module = get_conv_simple_script_module()
img = np.random.normal(size=[1, 3, 224, 224])
input_name = "input0"
shape_list = [(input_name, img.shape)]
mod, params = relay.frontend.from_pytorch(conv_simple_script_module, shape_list)
print('\n\n**** original mod: \n', mod['main'])
with relay.quantize.qconfig(calibrate_mode="global_scale", global_scale=8.0):
mod = relay.quantize.quantize(mod, params)
print('\n\n**** quantized mod: ', mod['main'])
if __name__ == '__main__':
main()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] lsy643 closed pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
lsy643 closed pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] giuseros commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
giuseros commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r520632201
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
Hi @lsy643 ,
So I think I would need a bit more info about this.
You have a float Resnet model (is this in Relay) and you want to quantize it, but the first conv layer does not get quantized. Is this what is happening? Can you reproduce this with a small test case (like a small Relay network with a Convolution + other nodes) and add this to `test_pass_auto_quantize.py` ?
cc @anijain2305 who knows more than me about quantizing a model
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] lsy643 commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
lsy643 commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r520993062
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
@giuseros
Yes, you are right.
For example, the input relay module could be:
```
fn (%input0: Tensor[(1, 3, 224, 224), float32], %conv1.weight: Tensor[(64, 3, 7, 7), float32], %conv2.weight: Tensor[(128, 64, 3, 3), float32]) {
%0 = nn.conv2d(%input0, %conv1.weight, strides=[2, 2], padding=[3, 3, 3, 3], channels=64, kernel_size=[7, 7]);
%1 = nn.relu(%0);
%2 = nn.max_pool2d(%1, pool_size=[3, 3], strides=[2, 2], padding=[1, 1, 1, 1]);
%3 = nn.conv2d(%2, %conv2.weight, padding=[1, 1, 1, 1], channels=128, kernel_size=[3, 3]);
nn.relu(%3)
}
```
And the mod after quantize is:
```
fn (%input0: Tensor[(1, 3, 224, 224), float32]) -> Tensor[(1, 128, 56, 56), float32] {
%0 = nn.conv2d(%input0, meta[relay.Constant][0] /* ty=Tensor[(64, 3, 7, 7), float32] */, strides=[2, 2], padding=[3, 3, 3, 3], channels=64, kernel_size=[7, 7]) /* ty=Tensor[(1, 64, 112, 112), float32] */;
%1 = nn.relu(%0) /* ty=Tensor[(1, 64, 112, 112), float32] */;
%2 = nn.max_pool2d(%1, pool_size=[3, 3], strides=[2, 2], padding=[1, 1, 1, 1]) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%3 = annotation.stop_fusion(%2) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%4 = multiply(%3, 16f /* ty=float32 */) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%5 = round(%4) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%6 = clip(%5, a_min=-127f, a_max=127f) /* ty=Tensor[(1, 64, 56, 56), float32] */;
%7 = cast(%6, dtype="int8") /* ty=Tensor[(1, 64, 56, 56), int8] */;
%8 = nn.conv2d(%7, meta[relay.Constant][1] /* ty=Tensor[(128, 64, 3, 3), int8] */, padding=[1, 1, 1, 1], channels=128, kernel_size=[3, 3], out_dtype="int32") /* ty=Tensor[(1, 128, 56, 56), int32] */;
%9 = nn.relu(%8) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%10 = add(%9, 256 /* ty=int32 */) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%11 = right_shift(%10, 9 /* ty=int32 */) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%12 = clip(%11, a_min=-127f, a_max=127f) /* ty=Tensor[(1, 128, 56, 56), int32] */;
%13 = cast(%12, dtype="int8") /* ty=Tensor[(1, 128, 56, 56), int8] */;
%14 = annotation.stop_fusion(%13) /* ty=Tensor[(1, 128, 56, 56), int8] */;
%15 = cast(%14, dtype="float32") /* ty=Tensor[(1, 128, 56, 56), float32] */;
multiply(%15, 0.0625f /* ty=float32 */) /* ty=Tensor[(1, 128, 56, 56), float32] */
}
```
Clearly, you can see that the two `nn.conv2d` have different input type, weight type and output type. I believe this happens because in the `conv2d_partition_function` function, the `data_cond` from `partition_expr_check` is `False` for the first `Conv` Op.
I can provide a script to reproduce this issue, and you may need to install Pytorch and torchvision by `pip install torch==1.7.0` and `pip install torchvision==0.8.1`
```
import tvm
from tvm import relay
import torch
import torch.nn as nn
import torchvision
import numpy as np
class ConvSimple(nn.Module):
def __init__(self):
super(ConvSimple, self).__init__()
self.inplanes = 64
self.dilation = 1
norm_layer = nn.BatchNorm2d
self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=7, stride=2, padding=3, bias=False)
self.relu = nn.ReLU(inplace=True)
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
self.conv2 = nn.Conv2d(self.inplanes, 128, kernel_size=3, stride=1, padding=1,bias=False)
self.relu2 = nn.ReLU(inplace=True)
self.conv3 = nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1, bias=False)
self.relu3 = nn.ReLU(inplace=True)
self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
for m in self.modules():
if isinstance(m, nn.Conv2d):
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
def forward(self, x):
x = self.conv1(x)
x = self.relu(x)
x = self.maxpool(x)
x = self.conv2(x)
x = self.relu2(x)
return x
def get_conv_simple_script_module():
conv_simple = ConvSimple()
conv_simple.eval()
input_shape = [1, 3, 224, 224]
input_data = torch.randn(input_shape)
traced_conv_simple = torch.jit.trace(conv_simple, input_data).eval()
return traced_conv_simple
def main():
conv_simple_script_module = get_conv_simple_script_module()
img = np.random.normal(size=[1, 3, 224, 224])
input_name = "input0"
shape_list = [(input_name, img.shape)]
mod, params = relay.frontend.from_pytorch(conv_simple_script_module, shape_list)
print('\n\n**** original mod: \n', mod['main'])
with relay.quantize.qconfig(calibrate_mode="global_scale", global_scale=8.0):
qnn1_mod = relay.quantize.quantize(mod, params)
print('\n\n**** quantized mod: ', qnn1_mod['main'])
with relay.quantize.qconfig(calibrate_mode="global_scale", global_scale=8.0, partition_conversions="enabled"):
qnn2_mod = relay.quantize.quantize(mod, params)
print('quantize_inputs: ', qnn2_mod['quantize_inputs'])
print('quantized_main: ', qnn2_mod['quantized_main'])
print('dequantize_outputs: ', qnn2_mod['dequantize_outputs'])
if __name__ == '__main__':
main()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] lsy643 commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
lsy643 commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r521009413
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
@giuseros
By the way, after I synchronized TVM repo last night, just adding those two lines would not make the first `Conv` get quantized neither. I am digging into it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] lsy643 commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
lsy643 commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r521009413
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
@giuseros
I have figured it out. It has not so much about the annotation pass.
Using `with relay.quantize.qconfig(calibrate_mode="global_scale", global_scale=8.0, skip_conv_layers=[])` can make the first `Conv` works fine.
I am just curious why the default value for `skip_conv_layers` is `[0]`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] giuseros commented on a change in pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
giuseros commented on a change in pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#discussion_r519867264
##########
File path: tests/python/relay/test_pass_auto_quantize.py
##########
@@ -148,7 +148,7 @@ def verify_partition(mod, params):
assert (
len(unpartitioned_mod.get_global_vars()) == 1
), "unpartitioned module should only have one function"
- with relay.quantize.qconfig(**BASE_CFG, partition_conversions="fully_integral"):
+ with relay.quantize.qconfig(**BASE_CFG):
Review comment:
Just trying to understand this change. From what I read, the default `qconfig.partition_conversions` is `disabled`. Wouldn't this test be the same as the one on line 146?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] anijain2305 commented on pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
anijain2305 commented on pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#issuecomment-725731041
Sorry for the late reply. I think this is because of `skip_conv_layers`
By default, it skips the first conv layer. Maybe, because first layer has higher impact on accuracy. So, I dont think this is needed. You can use QConfig to set this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] zhiics commented on pull request #6864: Qnn for first conv
Posted by GitBox <gi...@apache.org>.
zhiics commented on pull request #6864:
URL: https://github.com/apache/incubator-tvm/pull/6864#issuecomment-725694233
cc @anijain2305 @ZihengJiang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org