You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/10/28 03:54:53 UTC
[GitHub] [incubator-tvm] masahi opened a new pull request #6782: Torch quant linear dynamic
masahi opened a new pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782
This adds support for what PyTorch calls "dynamic quantization", where weights are quantized ahead of time but activations are quantized on the fly at runtime. See more details in:
https://pytorch.org/blog/introduction-to-quantization-on-pytorch/#the-three-modes-of-quantization-supported-in-pytorch-starting-version-13
https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html
https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html
TVM doesn't support such quantization flow at the moment. This flow is in a sweep spot in terms of quantization easy of use and performance, so I think this is worth supporting. Here are pros/cons compared to the static quantization (the one we do support):
Pros:
* API is trivial and quantization is automatic. No need to rewrite a model or do calibration (which is required for other quantization workflow in PyTorch).
* Weight is quantized ahead of time, so model size becomes much smaller. We can also use int8 math.
Cons:
* Scale and zero point calculation is done at runtime, so there is some overhead compared to the more standard static quantization.
My motivation for introducing this flow is to support quantized models from `transformers` like BERT and GPT2, where dynamic quantization via PyTorch or ONNXRuntime is the only quantization path they support (from what I understand). See the following blog post and the accompanying notebook by the ONNXRuntime team for inspiration.
https://medium.com/microsoftazure/faster-and-smaller-quantized-nlp-with-hugging-face-and-onnx-runtime-ec5525473bb7
https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/quantization/notebooks/Bert-GLUE_OnnxRuntime_quantization.ipynb
This PR has changes required for supporting dynamic quantization flow via QNN.
* Support non constant qparams in QNN quantize and dense op
* Add Torchscript `quantized::linear_dynamic` op converter in PyTorch frontend.
I prepared [a script](https://github.com/masahi/torchscript-to-tvm/tree/master/transformers) to evaluate accuracy and performance of BERT quantized via dynamic quantization, and compiled by TVM. The accuracy is reasonable but performance is terrible. Even with MKL enabled, TVM int8 is 3-4x slower than PyTorch (I haven't looked into details). I sense a big oppotunity here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#discussion_r513775161
##########
File path: python/tvm/relay/frontend/qnn_torch.py
##########
@@ -826,6 +826,74 @@ def _impl(inputs, _):
return _impl
+def _linear_dynamic():
+ def _calculate_qparam(inp):
+ # reference ATen/native/quantized/cpu/qlinear_dynamic.cpp
+ # ChooseQuantizationParams function
+ mn = _op.min(inp)
+ mx = _op.max(inp)
+
+ # Ensure that the interval contains 0
+ mn = _op.minimum(mn, _op.const(0.0, dtype="float32"))
+ mx = _op.maximum(mx, _op.const(0.0, dtype="float32"))
+
+ qmax = 255
+
+ # reduce_range became True in v1.6
+ if is_version_greater_than("1.5.1"):
+ qmax = 127
+
+ scale = (mx - mn) / _expr.const(qmax, dtype="float32")
+
+ zero_point_from_min = -(mn / scale)
+ zero_point = _op.cast(_op.round(_op.clip(zero_point_from_min, 0.0, qmax)), "int32")
+
+ return scale, zero_point
+
+ def _impl(inputs, _):
+ weight = inputs[1][0]
+ weight_scale = inputs[1][1]
+ weight_zero_point = inputs[1][2]
+
+ inp = inputs[0]
+
+ input_scale, input_zero_point = _calculate_qparam(inp)
+ qinp = relay.qnn.op.quantize(inp, input_scale, input_zero_point, out_dtype="uint8")
+
+ data_shape = infer_shape(inp)
+
+ if len(data_shape) > 2:
+ qinp = _op.reverse_reshape(qinp, [-1, 0])
+
+ weight_shape = infer_shape(weight)
+ units = weight_shape[0]
+ dense = relay.qnn.op.dense(
+ qinp,
+ weight,
+ input_zero_point,
+ weight_zero_point,
+ input_scale,
+ weight_scale,
+ units=units,
+ )
+ bias_var = inputs[1][3]
+
+ dequant_scale = input_scale * weight_scale
+ dense_out = _op.cast(dense, "float32") * dequant_scale
Review comment:
It might be better to call qnn.dequantize operation here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] masahi commented on a change in pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#discussion_r513790693
##########
File path: python/tvm/relay/frontend/qnn_torch.py
##########
@@ -826,6 +826,74 @@ def _impl(inputs, _):
return _impl
+def _linear_dynamic():
+ def _calculate_qparam(inp):
+ # reference ATen/native/quantized/cpu/qlinear_dynamic.cpp
+ # ChooseQuantizationParams function
+ mn = _op.min(inp)
+ mx = _op.max(inp)
+
+ # Ensure that the interval contains 0
+ mn = _op.minimum(mn, _op.const(0.0, dtype="float32"))
+ mx = _op.maximum(mx, _op.const(0.0, dtype="float32"))
+
+ qmax = 255
+
+ # reduce_range became True in v1.6
+ if is_version_greater_than("1.5.1"):
+ qmax = 127
+
+ scale = (mx - mn) / _expr.const(qmax, dtype="float32")
+
+ zero_point_from_min = -(mn / scale)
+ zero_point = _op.cast(_op.round(_op.clip(zero_point_from_min, 0.0, qmax)), "int32")
+
+ return scale, zero_point
+
+ def _impl(inputs, _):
+ weight = inputs[1][0]
+ weight_scale = inputs[1][1]
+ weight_zero_point = inputs[1][2]
+
+ inp = inputs[0]
+
+ input_scale, input_zero_point = _calculate_qparam(inp)
+ qinp = relay.qnn.op.quantize(inp, input_scale, input_zero_point, out_dtype="uint8")
+
+ data_shape = infer_shape(inp)
+
+ if len(data_shape) > 2:
+ qinp = _op.reverse_reshape(qinp, [-1, 0])
+
+ weight_shape = infer_shape(weight)
+ units = weight_shape[0]
+ dense = relay.qnn.op.dense(
+ qinp,
+ weight,
+ input_zero_point,
+ weight_zero_point,
+ input_scale,
+ weight_scale,
+ units=units,
+ )
+ bias_var = inputs[1][3]
+
+ dequant_scale = input_scale * weight_scale
+ dense_out = _op.cast(dense, "float32") * dequant_scale
Review comment:
ok replaced with dequantize, thanks for the thought!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] masahi merged pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
masahi merged pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] masahi commented on a change in pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#discussion_r513778997
##########
File path: python/tvm/relay/frontend/qnn_torch.py
##########
@@ -826,6 +826,74 @@ def _impl(inputs, _):
return _impl
+def _linear_dynamic():
+ def _calculate_qparam(inp):
+ # reference ATen/native/quantized/cpu/qlinear_dynamic.cpp
+ # ChooseQuantizationParams function
+ mn = _op.min(inp)
+ mx = _op.max(inp)
+
+ # Ensure that the interval contains 0
+ mn = _op.minimum(mn, _op.const(0.0, dtype="float32"))
+ mx = _op.maximum(mx, _op.const(0.0, dtype="float32"))
+
+ qmax = 255
+
+ # reduce_range became True in v1.6
+ if is_version_greater_than("1.5.1"):
+ qmax = 127
+
+ scale = (mx - mn) / _expr.const(qmax, dtype="float32")
+
+ zero_point_from_min = -(mn / scale)
+ zero_point = _op.cast(_op.round(_op.clip(zero_point_from_min, 0.0, qmax)), "int32")
+
+ return scale, zero_point
+
+ def _impl(inputs, _):
+ weight = inputs[1][0]
+ weight_scale = inputs[1][1]
+ weight_zero_point = inputs[1][2]
+
+ inp = inputs[0]
+
+ input_scale, input_zero_point = _calculate_qparam(inp)
+ qinp = relay.qnn.op.quantize(inp, input_scale, input_zero_point, out_dtype="uint8")
+
+ data_shape = infer_shape(inp)
+
+ if len(data_shape) > 2:
+ qinp = _op.reverse_reshape(qinp, [-1, 0])
+
+ weight_shape = infer_shape(weight)
+ units = weight_shape[0]
+ dense = relay.qnn.op.dense(
+ qinp,
+ weight,
+ input_zero_point,
+ weight_zero_point,
+ input_scale,
+ weight_scale,
+ units=units,
+ )
+ bias_var = inputs[1][3]
+
+ dequant_scale = input_scale * weight_scale
+ dense_out = _op.cast(dense, "float32") * dequant_scale
Review comment:
since I know zp is zero, I thought this should be a bit more efficient (doesn't need to add zp, not sure if qnn dequantize does this optimization)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] masahi commented on pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#issuecomment-718220692
> Overall looks good. Is this enough to run qBERT? I am surprised that we dont need to work on requantize here
Yes this is enough. Dynamic quantization flow replaces fp32 dense with runtime qparam calculation + int8 dense , leaving everything else fp32.
The output of `linear_dynamic` op is fp32, so I don't think we need requantize. PyTorch just cast int32 output to fp32, and multiply by input and weight scales, which I followed here. The corresponding implementation is here
https://github.com/pytorch/FBGEMM/blob/master/include/fbgemm/OutputProcessing-inl.h#L232
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#discussion_r513777765
##########
File path: python/tvm/relay/frontend/qnn_torch.py
##########
@@ -826,6 +826,74 @@ def _impl(inputs, _):
return _impl
+def _linear_dynamic():
+ def _calculate_qparam(inp):
+ # reference ATen/native/quantized/cpu/qlinear_dynamic.cpp
+ # ChooseQuantizationParams function
+ mn = _op.min(inp)
+ mx = _op.max(inp)
+
+ # Ensure that the interval contains 0
+ mn = _op.minimum(mn, _op.const(0.0, dtype="float32"))
+ mx = _op.maximum(mx, _op.const(0.0, dtype="float32"))
+
+ qmax = 255
+
+ # reduce_range became True in v1.6
+ if is_version_greater_than("1.5.1"):
+ qmax = 127
+
+ scale = (mx - mn) / _expr.const(qmax, dtype="float32")
+
+ zero_point_from_min = -(mn / scale)
+ zero_point = _op.cast(_op.round(_op.clip(zero_point_from_min, 0.0, qmax)), "int32")
+
+ return scale, zero_point
+
+ def _impl(inputs, _):
+ weight = inputs[1][0]
+ weight_scale = inputs[1][1]
+ weight_zero_point = inputs[1][2]
+
+ inp = inputs[0]
+
+ input_scale, input_zero_point = _calculate_qparam(inp)
+ qinp = relay.qnn.op.quantize(inp, input_scale, input_zero_point, out_dtype="uint8")
+
+ data_shape = infer_shape(inp)
+
+ if len(data_shape) > 2:
+ qinp = _op.reverse_reshape(qinp, [-1, 0])
+
+ weight_shape = infer_shape(weight)
+ units = weight_shape[0]
+ dense = relay.qnn.op.dense(
+ qinp,
+ weight,
+ input_zero_point,
+ weight_zero_point,
+ input_scale,
+ weight_scale,
+ units=units,
+ )
+ bias_var = inputs[1][3]
+
+ dequant_scale = input_scale * weight_scale
+ dense_out = _op.cast(dense, "float32") * dequant_scale
Review comment:
First readability, since we are working with qnn ops, one can easily understand that we are going from int to float domain just by seeing dequantize op.
Second, dequantize becomes one place where we handle per-tensor and per-channel quantization scales and non-zero zero points. Internally, dequantize will lower to the same exact operations that you have currently in the parser. In future, when we support per-channel scales for dense, we can just rely on dequantize op.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] anijain2305 commented on pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
anijain2305 commented on pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#issuecomment-718221534
> > Overall looks good. Is this enough to run qBERT? I am surprised that we dont need to work on requantize here
>
> Yes this is enough. Dynamic quantization flow replaces fp32 dense with runtime qparam calculation + int8 dense , leaving everything else fp32.
>
> The output of `linear_dynamic` op is fp32, so I don't think we need requantize. PyTorch just cast int32 output to fp32, and multiply by input and weight scales, which I followed here. The corresponding implementation is here
> https://github.com/pytorch/FBGEMM/blob/master/include/fbgemm/OutputProcessing-inl.h#L232
I see, what you describe is `dequantize` operation with `out_scale = input_scale * weigh_scale` and `out_zero_point=0`. This means that there will be lots of quantize and dequantize ops in the graph. That explains why requantize never showed up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#discussion_r513780334
##########
File path: python/tvm/relay/frontend/qnn_torch.py
##########
@@ -826,6 +826,74 @@ def _impl(inputs, _):
return _impl
+def _linear_dynamic():
+ def _calculate_qparam(inp):
+ # reference ATen/native/quantized/cpu/qlinear_dynamic.cpp
+ # ChooseQuantizationParams function
+ mn = _op.min(inp)
+ mx = _op.max(inp)
+
+ # Ensure that the interval contains 0
+ mn = _op.minimum(mn, _op.const(0.0, dtype="float32"))
+ mx = _op.maximum(mx, _op.const(0.0, dtype="float32"))
+
+ qmax = 255
+
+ # reduce_range became True in v1.6
+ if is_version_greater_than("1.5.1"):
+ qmax = 127
+
+ scale = (mx - mn) / _expr.const(qmax, dtype="float32")
+
+ zero_point_from_min = -(mn / scale)
+ zero_point = _op.cast(_op.round(_op.clip(zero_point_from_min, 0.0, qmax)), "int32")
+
+ return scale, zero_point
+
+ def _impl(inputs, _):
+ weight = inputs[1][0]
+ weight_scale = inputs[1][1]
+ weight_zero_point = inputs[1][2]
+
+ inp = inputs[0]
+
+ input_scale, input_zero_point = _calculate_qparam(inp)
+ qinp = relay.qnn.op.quantize(inp, input_scale, input_zero_point, out_dtype="uint8")
+
+ data_shape = infer_shape(inp)
+
+ if len(data_shape) > 2:
+ qinp = _op.reverse_reshape(qinp, [-1, 0])
+
+ weight_shape = infer_shape(weight)
+ units = weight_shape[0]
+ dense = relay.qnn.op.dense(
+ qinp,
+ weight,
+ input_zero_point,
+ weight_zero_point,
+ input_scale,
+ weight_scale,
+ units=units,
+ )
+ bias_var = inputs[1][3]
+
+ dequant_scale = input_scale * weight_scale
+ dense_out = _op.cast(dense, "float32") * dequant_scale
Review comment:
We should add that in qnn dequantize. It should be easy. (I just checked, I missed that optimization, it exists in other ops like conv2d and dense)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] masahi commented on a change in pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#discussion_r513771153
##########
File path: python/tvm/relay/frontend/qnn_torch.py
##########
@@ -826,6 +826,74 @@ def _impl(inputs, _):
return _impl
+def _linear_dynamic():
+ def _calculate_qparam(inp):
+ # reference ATen/native/quantized/cpu/qlinear_dynamic.cpp
+ # ChooseQuantizationParams function
+ mn = _op.min(inp)
+ mx = _op.max(inp)
+
+ # Ensure that the interval contains 0
+ mn = _op.minimum(mn, _op.const(0.0, dtype="float32"))
+ mx = _op.maximum(mx, _op.const(0.0, dtype="float32"))
+
+ qmax = 255
+
+ # reduce_range became True in v1.6
+ if is_version_greater_than("1.5.1"):
+ qmax = 127
Review comment:
This comes from https://github.com/pytorch/pytorch/blob/d642992877139671466d2a96663abede9e39ad55/aten/src/ATen/native/quantized/cpu/quant_utils.h#L64-L66
Here, they intentionally reduce the possible range of quantized values by half, i.e. [qmin, qmax] to [qmin/2, qmax/2]. Since PyTorch only uses uint8, this is fine.
It's not clear to me why they do this, but the following PR has some explanation: "reduce_range option restricts the activation tensor to 7 bits instead of 8.This is necessary to enable per channel quant for RNNs and LSTMs" https://github.com/pytorch/pytorch/pull/39041
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] masahi commented on a change in pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#discussion_r513778997
##########
File path: python/tvm/relay/frontend/qnn_torch.py
##########
@@ -826,6 +826,74 @@ def _impl(inputs, _):
return _impl
+def _linear_dynamic():
+ def _calculate_qparam(inp):
+ # reference ATen/native/quantized/cpu/qlinear_dynamic.cpp
+ # ChooseQuantizationParams function
+ mn = _op.min(inp)
+ mx = _op.max(inp)
+
+ # Ensure that the interval contains 0
+ mn = _op.minimum(mn, _op.const(0.0, dtype="float32"))
+ mx = _op.maximum(mx, _op.const(0.0, dtype="float32"))
+
+ qmax = 255
+
+ # reduce_range became True in v1.6
+ if is_version_greater_than("1.5.1"):
+ qmax = 127
+
+ scale = (mx - mn) / _expr.const(qmax, dtype="float32")
+
+ zero_point_from_min = -(mn / scale)
+ zero_point = _op.cast(_op.round(_op.clip(zero_point_from_min, 0.0, qmax)), "int32")
+
+ return scale, zero_point
+
+ def _impl(inputs, _):
+ weight = inputs[1][0]
+ weight_scale = inputs[1][1]
+ weight_zero_point = inputs[1][2]
+
+ inp = inputs[0]
+
+ input_scale, input_zero_point = _calculate_qparam(inp)
+ qinp = relay.qnn.op.quantize(inp, input_scale, input_zero_point, out_dtype="uint8")
+
+ data_shape = infer_shape(inp)
+
+ if len(data_shape) > 2:
+ qinp = _op.reverse_reshape(qinp, [-1, 0])
+
+ weight_shape = infer_shape(weight)
+ units = weight_shape[0]
+ dense = relay.qnn.op.dense(
+ qinp,
+ weight,
+ input_zero_point,
+ weight_zero_point,
+ input_scale,
+ weight_scale,
+ units=units,
+ )
+ bias_var = inputs[1][3]
+
+ dequant_scale = input_scale * weight_scale
+ dense_out = _op.cast(dense, "float32") * dequant_scale
Review comment:
since I know zp is zero, I thought this should be a bit more efficient
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] masahi commented on pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#issuecomment-718358510
Thanks @anijain2305
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] masahi commented on a change in pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#discussion_r513775708
##########
File path: python/tvm/relay/frontend/qnn_torch.py
##########
@@ -826,6 +826,74 @@ def _impl(inputs, _):
return _impl
+def _linear_dynamic():
+ def _calculate_qparam(inp):
+ # reference ATen/native/quantized/cpu/qlinear_dynamic.cpp
+ # ChooseQuantizationParams function
+ mn = _op.min(inp)
+ mx = _op.max(inp)
+
+ # Ensure that the interval contains 0
+ mn = _op.minimum(mn, _op.const(0.0, dtype="float32"))
+ mx = _op.maximum(mx, _op.const(0.0, dtype="float32"))
+
+ qmax = 255
+
+ # reduce_range became True in v1.6
+ if is_version_greater_than("1.5.1"):
+ qmax = 127
+
+ scale = (mx - mn) / _expr.const(qmax, dtype="float32")
+
+ zero_point_from_min = -(mn / scale)
+ zero_point = _op.cast(_op.round(_op.clip(zero_point_from_min, 0.0, qmax)), "int32")
+
+ return scale, zero_point
+
+ def _impl(inputs, _):
+ weight = inputs[1][0]
+ weight_scale = inputs[1][1]
+ weight_zero_point = inputs[1][2]
+
+ inp = inputs[0]
+
+ input_scale, input_zero_point = _calculate_qparam(inp)
+ qinp = relay.qnn.op.quantize(inp, input_scale, input_zero_point, out_dtype="uint8")
+
+ data_shape = infer_shape(inp)
+
+ if len(data_shape) > 2:
+ qinp = _op.reverse_reshape(qinp, [-1, 0])
+
+ weight_shape = infer_shape(weight)
+ units = weight_shape[0]
+ dense = relay.qnn.op.dense(
+ qinp,
+ weight,
+ input_zero_point,
+ weight_zero_point,
+ input_scale,
+ weight_scale,
+ units=units,
+ )
+ bias_var = inputs[1][3]
+
+ dequant_scale = input_scale * weight_scale
+ dense_out = _op.cast(dense, "float32") * dequant_scale
Review comment:
why better? I did try dequantize but it didn't matter in terms of accuracy
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#discussion_r513723931
##########
File path: python/tvm/relay/frontend/qnn_torch.py
##########
@@ -826,6 +826,74 @@ def _impl(inputs, _):
return _impl
+def _linear_dynamic():
+ def _calculate_qparam(inp):
+ # reference ATen/native/quantized/cpu/qlinear_dynamic.cpp
+ # ChooseQuantizationParams function
+ mn = _op.min(inp)
+ mx = _op.max(inp)
+
+ # Ensure that the interval contains 0
+ mn = _op.minimum(mn, _op.const(0.0, dtype="float32"))
+ mx = _op.maximum(mx, _op.const(0.0, dtype="float32"))
+
+ qmax = 255
+
+ # reduce_range became True in v1.6
+ if is_version_greater_than("1.5.1"):
+ qmax = 127
Review comment:
Whats happening here? Should this be coupled with dtype - 127 should be for int8 while 255 for uint8
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] anijain2305 edited a comment on pull request #6782: [Torch, QNN] Support dynamic quantization flow to enable importing quantized transformer models
Posted by GitBox <gi...@apache.org>.
anijain2305 edited a comment on pull request #6782:
URL: https://github.com/apache/incubator-tvm/pull/6782#issuecomment-718221534
> > Overall looks good. Is this enough to run qBERT? I am surprised that we dont need to work on requantize here
>
> Yes this is enough. Dynamic quantization flow replaces fp32 dense with runtime qparam calculation + int8 dense , leaving everything else fp32.
>
> The output of `linear_dynamic` op is fp32, so I don't think we need requantize. PyTorch just cast int32 output to fp32, and multiply by input and weight scales, which I followed here. The corresponding implementation is here
> https://github.com/pytorch/FBGEMM/blob/master/include/fbgemm/OutputProcessing-inl.h#L232
I see, what you describe is `dequantize` operation with `in_scale = input_scale * weigh_scale` and `in_zero_point=0`. This means that there will be lots of quantize and dequantize ops in the graph. That explains why requantize never showed up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org