You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/06/18 23:24:36 UTC

[GitHub] [incubator-tvm] anijain2305 opened a new pull request #5848: [TFLite] TFLite 2.x parser quantization support.

anijain2305 opened a new pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848


   This is by no way ready (lots of hack and duplication currently). This is to kick start the discussion for TFLite2.x quantization model ingestion
   
   Checked on MNIST model - https://www.tensorflow.org/lite/performance/post_training_integer_quant
   
   @u99127 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r445827227



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -2566,17 +2613,27 @@ def convert_quantize(self, op):
         input_tensors = self.get_input_tensors(op)
         assert len(input_tensors) == 1, "input tensors length should be 1"
         input_tensor = input_tensors[0]
+        input_tensor_type_str = self.get_tensor_type_str(input_tensor.tensor.Type())
         in_expr = self.get_expr(input_tensor.tensor_idx)
 
         output_tensors = self.get_output_tensors(op)
         assert len(output_tensors) == 1, "output tensors length should be 1"
         output_tensor = output_tensors[0]
+        output_tensor_type_str = self.get_tensor_type_str(output_tensor.tensor.Type())
 
         # The output must be quantized
         assert output_tensor.qnn_params
-        # Quantize the input
-        out = self.quantize(in_expr, output_tensor)
 
+        # TFLite Quantize op can also act as Requantize op
+        if input_tensor_type_str == "float32":
+            out = self.quantize(in_expr, output_tensor)
+        else:
+            out = _qnn.op.requantize(in_expr,
+                                     input_scale=input_tensor.qnn_params['scale'],
+                                     input_zero_point=input_tensor.qnn_params['zero_point'],
+                                     output_scale=output_tensor.qnn_params['scale'],
+                                     output_zero_point=output_tensor.qnn_params['zero_point'],
+                                     out_dtype=output_tensor_type_str)
         return out

Review comment:
       ![image](https://user-images.githubusercontent.com/13822661/85793474-ebec7980-b6e9-11ea-8e53-fa3121ba50c0.png)
   
   Above test case added to force both types of quantize nodes




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r445355375



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -262,21 +298,25 @@ def get_tensor_value(self, tensor_wrapper):
         except ImportError:
             raise ImportError("The tflite package must be installed")
 
+        data = tensor_wrapper.buffer.DataAsNumpy()
+        shape = tensor_wrapper.tensor.ShapeAsNumpy()
+
+        # Set shape to 1 if the data is a scalar type
+        if data.shape == (1,) and isinstance(shape, int) and shape == 0:

Review comment:
       I modified the comments. Can you take a look now? This is sort of a corner case when the TFLite buffer is a scalar.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] siju-samuel commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
siju-samuel commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r447554160



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -651,12 +701,43 @@ def convert_shape(self, op):
 
     def convert_relu(self, op):
         """Convert TFLite ReLU"""
+        try:
+            from tflite.ActivationFunctionType import ActivationFunctionType
+        except ImportError:
+            raise ImportError("The tflite package must be installed")
+
         input_tensors = self.get_input_tensors(op)
         assert len(input_tensors) == 1, "input tensors length should be 1"
-
         input_tensor = input_tensors[0]
         in_expr = self.get_expr(input_tensor.tensor_idx)
-        out = _op.nn.relu(in_expr)
+
+        output_tensors = self.get_output_tensors(op)
+        assert len(output_tensors) == 1, "output tensors length should be 1"
+        output_tensor = output_tensors[0]
+
+        if input_tensor.qnn_params:
+            # Quantize a float value to an quantized integer value
+            scale_val = get_scalar_from_constant(input_tensor.qnn_params['scale'])
+            zero_point_val = get_scalar_from_constant(input_tensor.qnn_params['zero_point'])
+
+            output_tensor_type_str = self.get_tensor_type_str(output_tensor.tensor.Type())
+            out = self.convert_qnn_fused_activation_function(\
+                    expr=in_expr,
+                    fused_activation_fn=ActivationFunctionType.RELU,
+                    scale=scale_val,
+                    zero_point=zero_point_val,
+                    dtype=output_tensor_type_str)
+        else:
+            out = _op.clip(in_expr, a_min=0, a_max=6)

Review comment:
       This should be relu, not clip

##########
File path: tests/python/frontend/tflite/test_forward.py
##########
@@ -1686,38 +1812,33 @@ def test_forward_squeeze():
 def _test_quantize_dequantize(data):
     """ One iteration of quantize and dequantize """
 
-    # Define a dummy model
+    # Keras model to force TFLite converter to insert 2 TFLite quantize ops.
+    # First TFLite quantize op converts float32 tensor to int8 tensor - Qnn quantize.
+    # Second TLite quantize op converts int8 tensor to int8 tensor - Qnn requantize.

Review comment:
       TLite -> TFLite




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#issuecomment-652479286


   @u99127 ping


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] siju-samuel commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
siju-samuel commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r445310994



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -262,21 +298,25 @@ def get_tensor_value(self, tensor_wrapper):
         except ImportError:
             raise ImportError("The tflite package must be installed")
 
+        data = tensor_wrapper.buffer.DataAsNumpy()
+        shape = tensor_wrapper.tensor.ShapeAsNumpy()
+
+        # Set shape to 1 if the data is a scalar type
+        if data.shape == (1,) and isinstance(shape, int) and shape == 0:
+            shape = (1,)
+
+        if tensor_wrapper.tensor.Type() == TensorType.INT8:
+            return np.frombuffer(data, dtype=np.int8).reshape(shape)
         if tensor_wrapper.tensor.Type() == TensorType.UINT8:
-            return np.frombuffer(tensor_wrapper.buffer.DataAsNumpy(), dtype=np.uint8).reshape(
-                tensor_wrapper.tensor.ShapeAsNumpy())
-        if tensor_wrapper.tensor.Type() == TensorType.FLOAT32:
-            return np.frombuffer(tensor_wrapper.buffer.DataAsNumpy(), dtype=np.float32).reshape(
-                tensor_wrapper.tensor.ShapeAsNumpy())
+            return np.frombuffer(data, dtype=np.uint8).reshape(shape)

Review comment:
       Can we add support for int16/float16 as well?

##########
File path: tests/python/frontend/tflite/test_forward.py
##########
@@ -2445,6 +2467,112 @@ def test_forward_qnn_mobilenet_v3_net():
     tvm.testing.assert_allclose(tvm_sorted_labels, tflite_sorted_labels)
 
 
+
+def _quantize_tf_hub_keras_model(url, height, width):
+    keras_model = tf.keras.Sequential([hub.KerasLayer(url, output_shape=[1001])])
+    data = pre_processed_image(height, width)
+
+    # Set the input shapes of the keras model
+    keras_model._set_inputs(data)
+
+    # Get the converter
+    converter = interpreter_wrapper.TFLiteConverter.from_keras_model(keras_model)
+
+    # To create quantized values with dynamic range of activations, needs representative dataset
+    def representative_data_gen():
+        for i in range(1):
+            yield [data]
+
+    converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
+    converter.representative_dataset = representative_data_gen
+    return converter.convert()
+
+
+def test_forward_tflite2_qnn_resnet50():
+    """Test the Quantized TFLite version 2.1.0 Resnet50 model."""
+    if package_version.parse(tf.VERSION) >= package_version.parse('2.1.0'):
+        # Quantize the model
+        url = "https://tfhub.dev/tensorflow/resnet_50/classification/1"
+        tflite_model_buf = _quantize_tf_hub_keras_model(url, 224, 224)
+        data = pre_processed_image(224, 224)
+
+        tflite_output = run_tflite_graph(tflite_model_buf, data)
+        tflite_predictions = np.squeeze(tflite_output)
+        tflite_sorted_labels = tflite_predictions.argsort()[-3:][::-1]
+        tvm_output = run_tvm_graph(tflite_model_buf, np.array(data), 'input_1')
+        tvm_predictions = np.squeeze(tvm_output)
+        tvm_sorted_labels = tvm_predictions.argsort()[-3:][::-1]
+        tvm.testing.assert_allclose(tvm_sorted_labels, tflite_sorted_labels)

Review comment:
       just for info, How different is the output values? if comparable, we can do the output value comparison itself. Since rounding method is different in tflite and tvm and due to that there are some difference in  outputs.

##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -1605,7 +1645,7 @@ def convert_fully_connected(self, op):
 
         # weight tensor type should be UINT8 (quantization) or FLOAT32
         weight_tensor_type = weight_tensor.tensor.Type()
-        assert weight_tensor_type in (TensorType.UINT8, TensorType.FLOAT32)
+        assert weight_tensor_type in (TensorType.INT8, TensorType.UINT8, TensorType.FLOAT32)

Review comment:
       change the comment in line 1646 as well to include INT8. check similar in other places also

##########
File path: tests/python/frontend/tflite/test_forward.py
##########
@@ -2445,6 +2467,112 @@ def test_forward_qnn_mobilenet_v3_net():
     tvm.testing.assert_allclose(tvm_sorted_labels, tflite_sorted_labels)
 
 
+
+def _quantize_tf_hub_keras_model(url, height, width):

Review comment:
       Add func headers similar to previous testcases.

##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -2723,6 +2779,15 @@ def get_scalar_from_constant(expr):
         "value must be float32/int32"
     return np.asscalar(value)
 
+def get_vector_from_constant(expr):
+    """ Returns scalar value from Relay constant scalar. """

Review comment:
       Update docstring

##########
File path: tests/python/frontend/tflite/test_forward.py
##########
@@ -73,6 +74,28 @@ def get_real_image(im_height, im_width):
     data = np.reshape(x, (1, im_height, im_width, 3))
     return data
 
+
+def pre_processed_image(height, width):
+    repo_base = 'https://github.com/dmlc/web-data/raw/master/tensorflow/models/InceptionV1/'
+    img_name = 'elephant-299.jpg'
+    image_url = os.path.join(repo_base, img_name)
+    img_path = download_testdata(image_url, img_name, module='data')
+    image = tf.io.read_file(img_path)
+    image = tf.image.decode_jpeg(image, channels=3)
+    with tf.name_scope('eval_image'):
+        if image.dtype != tf.float32:
+            image = tf.image.convert_image_dtype(image, dtype=tf.float32)
+        image = tf.image.central_crop(image, central_fraction=0.875)
+    # Resize the image to the specified height and width.
+    image = tf.expand_dims(image, 0)
+    image = tf.image.resize(image, [height, width],
+                            align_corners=False)
+    image = tf.image.resize(image, [height, width])
+    image = tf.squeeze(image, [0])
+    image = tf.expand_dims(image, axis=0)

Review comment:
        i think this squeeze and expanddims are no reqiuired since both are done at axis 0. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#issuecomment-651205434


   @u99127 @siju-samuel CI passes now. Can you please take another look? There were more changes, mostly to add unit level tests.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] u99127 commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
u99127 commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r448548518



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -651,12 +701,43 @@ def convert_shape(self, op):
 
     def convert_relu(self, op):
         """Convert TFLite ReLU"""
+        try:
+            from tflite.ActivationFunctionType import ActivationFunctionType
+        except ImportError:
+            raise ImportError("The tflite package must be installed")
+

Review comment:
       I think this is unnecessary given the import of ActivationFunctionType in the constructor [here](https://github.com/apache/incubator-tvm/blob/b979bf6a7630ada055fff2d65e1cd0f8d55bb6a0/python/tvm/relay/frontend/tflite.py#L54)
   

##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -692,6 +773,11 @@ def _hard_swish(data):
 
     def convert_relu6(self, op):
         """Convert TFLite ReLU6"""
+        try:
+            from tflite.ActivationFunctionType import ActivationFunctionType

Review comment:
       Same as relu,  I think this is unnecessary given the import of ActivationFunctionType in the constructor [here](https://github.com/apache/incubator-tvm/blob/b979bf6a7630ada055fff2d65e1cd0f8d55bb6a0/python/tvm/relay/frontend/tflite.py#L54)

##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -263,21 +305,29 @@ def get_tensor_value(self, tensor_wrapper):
         except ImportError:
             raise ImportError("The tflite package must be installed")
 
+        # Read the data from the buffer. Also extract the shape.
+        # The shape is used later to reshape the data.
+        data = tensor_wrapper.buffer.DataAsNumpy()
+        shape = tensor_wrapper.tensor.ShapeAsNumpy()
+
+        # When TFLite buffer is of size 1 (scalar), then TFLite tensor shape is set to 0.
+        # Therefore, we set the shape to 1 for numpy reshape to work.  Set shape to 1 if the data is
+        # a scalar type
+        if data.size == 1 and isinstance(shape, int) and shape == 0:
+            shape = (1,)
+
+        if tensor_wrapper.tensor.Type() == TensorType.INT8:

Review comment:
       Minor nit and this should really be credited to Dmitriy Smirnov. https://github.com/d-smirnov
   
   the condition here could well be pulled out into a helper function that has a dictionary to help us map from TensorType to numpy type ? 
   
   Would make the code much cleaner and reduce duplication. 
   
   i.e. something like 
   
   
   def get_tensor_type_as_numpy(self, tensor_wrapper):
           """Returns np.dtype out of TensorType"""
           """Returns np.dtype out of TensorType"""
           assert isinstance(tensor_wrapper, TensorWrapper)
   
           try:
               from tflite.TensorType import TensorType
               return { TensorType.UINT8:   np.uint8,
                        TensorType.INT8:    np.int8,   
                        TensorType.FLOAT32: np.float32,
                        TensorType.INT32:   np.int32,
                        TensorType.INT64:   np.int64,
                        TensorType.BOOL:    np.bool_ }[ tensor_wrapper.tensor.Type() ]
           except ImportError:
               raise ImportError("The tflite package must be installed")
   
           except KeyError:
               raise NotImplementedError("Tensor type '{}' currently not supported"
                                         .format(tensor_wrapper.tensor.Type()))
   
       def get_tensor_value(self, tensor_wrapper):
           """Get tensor buffer value from given tensor wrapper"""
           assert isinstance(tensor_wrapper, TensorWrapper)
   
           value_type = self.get_tensor_type_as_numpy( tensor_wrapper )
           return np.frombuffer( tensor_wrapper.buffer.DataAsNumpy(), dtype=value_type ).reshape(
                   tensor_wrapper.tensor.ShapeAsNumpy() \
                       if 0 != tensor_wrapper.tensor.ShapeLength() \
                           else [] )
   
   

##########
File path: tests/python/frontend/tflite/test_forward.py
##########
@@ -787,14 +897,30 @@ def test_forward_convolution():
         _test_convolution([4, 17, 17, 124], [1, 1, 124, 19], [1, 1], [1, 1], 'SAME', 'NHWC', quantized=quantized)
         _test_convolution([4, 17, 17, 12], [3, 3, 12, 32], [1, 1], [2, 2], 'VALID', 'NHWC', quantized=quantized)
 
-    # depthwise convolution
-    _test_convolution([4, 8, 8, 176], [1, 1, 176, 1], [1, 1], [1, 1], 'SAME', 'NHWC', True)
-    _test_convolution([4, 17, 17, 19], [3, 3, 19, 1], [1, 1], [2, 2], 'VALID', 'NHWC', True)
-    _test_convolution([4, 17, 17, 124], [1, 1, 124, 1], [1, 1], [1, 1], 'SAME', 'NHWC', True)
-    _test_convolution([4, 17, 17, 12], [3, 3, 12, 1], [1, 1], [2, 2], 'VALID', 'NHWC', True)
-    _test_convolution([4, 17, 17, 12], [3, 3, 12, 2], [1, 1], [2, 2], 'VALID', 'NHWC', True)
-    # dephtwise convolution with single input channel
-    _test_convolution([1, 76, 64, 1], [9, 5, 1, 96], [1, 1], [1, 1], 'SAME', 'NHWC', True)
+        # depthwise convolution
+        _test_convolution([4, 8, 8, 176], [1, 1, 176, 1], [1, 1], [1, 1], 'SAME', 'NHWC', True, quantized=quantized)
+        _test_convolution([4, 17, 17, 19], [3, 3, 19, 1], [1, 1], [2, 2], 'VALID', 'NHWC', True, quantized=quantized)
+        _test_convolution([4, 17, 17, 124], [1, 1, 124, 1], [1, 1], [1, 1], 'SAME', 'NHWC', True, quantized=quantized)
+        _test_convolution([4, 17, 17, 12], [3, 3, 12, 1], [1, 1], [2, 2], 'VALID', 'NHWC', True, quantized=quantized)
+        _test_convolution([4, 17, 17, 12], [3, 3, 12, 2], [1, 1], [2, 2], 'VALID', 'NHWC', True, quantized=quantized)
+        # dephtwise convolution with single input channel

Review comment:
       dephtwise / depthwise.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] siju-samuel commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
siju-samuel commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r445531482



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -258,15 +260,18 @@ def get_tensors(self, tensors_idx_list):
                     assert isinstance(tflite_zero_point, np.ndarray)
 
                     # Tensor - Per-axis quantization
-                    if tflite_scale.shape != (1,) and tflite_zero_point.shape != (1,):
+                    if tflite_scale.size != 1 and tflite_zero_point.size != 1:
                         scale = tflite_scale
-                        # Ensure that all zero points are identical
+                        # Ensure that all zero points are zeros
                         zero_point = tflite_zero_point
-                        assert all(x == zero_point[0] for x in zero_point)
+                        if not all(x == 0 for x in zero_point):
+                            raise tvm.error.OpAttributeInvalid(\
+                                    "TFLite per-axis quantization restricts all zero points to be"
+                                    + " 0, but a non-zero value is observed")
                         zero_point = int(zero_point[0])
 
                     # Scalar - Per-tensor quantization
-                    elif tflite_scale.shape == (1,) and tflite_zero_point.shape == (1,):
+                    elif tflite_scale.size == 1 and tflite_zero_point.size == 1:
                         scale = float(tflite_scale[0])
                         zero_point = int(tflite_zero_point[0])
 

Review comment:
       error coud be due to wrong `tflite_zero_point`as well,  plz print type of `tflite_zero_point` in else case.

##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -298,11 +303,15 @@ def get_tensor_value(self, tensor_wrapper):
         except ImportError:
             raise ImportError("The tflite package must be installed")
 
+        # Read the data from the buffer. Also extract the shape.
+        # The shape is used later to reshape the data.
         data = tensor_wrapper.buffer.DataAsNumpy()
         shape = tensor_wrapper.tensor.ShapeAsNumpy()
 
-        # Set shape to 1 if the data is a scalar type
-        if data.shape == (1,) and isinstance(shape, int) and shape == 0:
+        # When TFLite buffer is of size 1 (scalar), then TFLite tensor shape is set to 0.
+        # Therefore, we set the shape to 1 for numpy reshape to work.
+        Set shape to 1 if the data is a scalar type

Review comment:
       missed to add # before comment?

##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -258,15 +260,18 @@ def get_tensors(self, tensors_idx_list):
                     assert isinstance(tflite_zero_point, np.ndarray)
 
                     # Tensor - Per-axis quantization
-                    if tflite_scale.shape != (1,) and tflite_zero_point.shape != (1,):
+                    if tflite_scale.size != 1 and tflite_zero_point.size != 1:
                         scale = tflite_scale
-                        # Ensure that all zero points are identical
+                        # Ensure that all zero points are zeros
                         zero_point = tflite_zero_point
-                        assert all(x == zero_point[0] for x in zero_point)
+                        if not all(x == 0 for x in zero_point):

Review comment:
       Suggestion `if not all(x == 0 for x in zero_point):` >> `if not np.all((zero_point == 0)):`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r448593312



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -651,12 +701,43 @@ def convert_shape(self, op):
 
     def convert_relu(self, op):
         """Convert TFLite ReLU"""
+        try:
+            from tflite.ActivationFunctionType import ActivationFunctionType
+        except ImportError:
+            raise ImportError("The tflite package must be installed")
+

Review comment:
       Tried this but it failed, the scope of imports is limited to the functions in which they are imported.

##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -692,6 +773,11 @@ def _hard_swish(data):
 
     def convert_relu6(self, op):
         """Convert TFLite ReLU6"""
+        try:
+            from tflite.ActivationFunctionType import ActivationFunctionType

Review comment:
       Same as before




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r445326823



##########
File path: tests/python/frontend/tflite/test_forward.py
##########
@@ -2445,6 +2467,112 @@ def test_forward_qnn_mobilenet_v3_net():
     tvm.testing.assert_allclose(tvm_sorted_labels, tflite_sorted_labels)
 
 
+
+def _quantize_tf_hub_keras_model(url, height, width):
+    keras_model = tf.keras.Sequential([hub.KerasLayer(url, output_shape=[1001])])
+    data = pre_processed_image(height, width)
+
+    # Set the input shapes of the keras model
+    keras_model._set_inputs(data)
+
+    # Get the converter
+    converter = interpreter_wrapper.TFLiteConverter.from_keras_model(keras_model)
+
+    # To create quantized values with dynamic range of activations, needs representative dataset
+    def representative_data_gen():
+        for i in range(1):
+            yield [data]
+
+    converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
+    converter.representative_dataset = representative_data_gen
+    return converter.convert()
+
+
+def test_forward_tflite2_qnn_resnet50():
+    """Test the Quantized TFLite version 2.1.0 Resnet50 model."""
+    if package_version.parse(tf.VERSION) >= package_version.parse('2.1.0'):
+        # Quantize the model
+        url = "https://tfhub.dev/tensorflow/resnet_50/classification/1"
+        tflite_model_buf = _quantize_tf_hub_keras_model(url, 224, 224)
+        data = pre_processed_image(224, 224)
+
+        tflite_output = run_tflite_graph(tflite_model_buf, data)
+        tflite_predictions = np.squeeze(tflite_output)
+        tflite_sorted_labels = tflite_predictions.argsort()[-3:][::-1]
+        tvm_output = run_tvm_graph(tflite_model_buf, np.array(data), 'input_1')
+        tvm_predictions = np.squeeze(tvm_output)
+        tvm_sorted_labels = tvm_predictions.argsort()[-3:][::-1]
+        tvm.testing.assert_allclose(tvm_sorted_labels, tflite_sorted_labels)

Review comment:
       Yes, rounding differences lead to these differences (although minimal effect on Top1/5 accuracy). I tried softmax'd output for older TFLite 1.13 tests and labels looked better for testing. I will check the TFLite 2.1.0 tests (I dont think picture would change)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r445326022



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -262,21 +298,25 @@ def get_tensor_value(self, tensor_wrapper):
         except ImportError:
             raise ImportError("The tflite package must be installed")
 
+        data = tensor_wrapper.buffer.DataAsNumpy()
+        shape = tensor_wrapper.tensor.ShapeAsNumpy()
+
+        # Set shape to 1 if the data is a scalar type
+        if data.shape == (1,) and isinstance(shape, int) and shape == 0:
+            shape = (1,)
+
+        if tensor_wrapper.tensor.Type() == TensorType.INT8:
+            return np.frombuffer(data, dtype=np.int8).reshape(shape)
         if tensor_wrapper.tensor.Type() == TensorType.UINT8:
-            return np.frombuffer(tensor_wrapper.buffer.DataAsNumpy(), dtype=np.uint8).reshape(
-                tensor_wrapper.tensor.ShapeAsNumpy())
-        if tensor_wrapper.tensor.Type() == TensorType.FLOAT32:
-            return np.frombuffer(tensor_wrapper.buffer.DataAsNumpy(), dtype=np.float32).reshape(
-                tensor_wrapper.tensor.ShapeAsNumpy())
+            return np.frombuffer(data, dtype=np.uint8).reshape(shape)

Review comment:
       Samuel. Thats a good point. I looked into int16 support. This is experimental in TFLite for now. 
   
   For float16, I think we will need a good number of tests. I think we can handle float16 in a separate PR.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#issuecomment-653176255


   @siju-samuel Please approve and merge if its looks good :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#issuecomment-649036699


   Thanks @u99127 for the review. All make sense.  I will add clarifications.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r445135010



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -2566,17 +2613,27 @@ def convert_quantize(self, op):
         input_tensors = self.get_input_tensors(op)
         assert len(input_tensors) == 1, "input tensors length should be 1"
         input_tensor = input_tensors[0]
+        input_tensor_type_str = self.get_tensor_type_str(input_tensor.tensor.Type())
         in_expr = self.get_expr(input_tensor.tensor_idx)
 
         output_tensors = self.get_output_tensors(op)
         assert len(output_tensors) == 1, "output tensors length should be 1"
         output_tensor = output_tensors[0]
+        output_tensor_type_str = self.get_tensor_type_str(output_tensor.tensor.Type())
 
         # The output must be quantized
         assert output_tensor.qnn_params
-        # Quantize the input
-        out = self.quantize(in_expr, output_tensor)
 
+        # TFLite Quantize op can also act as Requantize op
+        if input_tensor_type_str == "float32":
+            out = self.quantize(in_expr, output_tensor)
+        else:
+            out = _qnn.op.requantize(in_expr,
+                                     input_scale=input_tensor.qnn_params['scale'],
+                                     input_zero_point=input_tensor.qnn_params['zero_point'],
+                                     output_scale=output_tensor.qnn_params['scale'],
+                                     output_zero_point=output_tensor.qnn_params['zero_point'],
+                                     out_dtype=output_tensor_type_str)
         return out

Review comment:
       You are right. I will add a test case in this PR. This will enable us to keep those 5 end to end tests as well.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] u99127 commented on pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
u99127 commented on pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#issuecomment-653135171


   > 
   > 
   > @u99127 can you please take a look again?
   
   LGTM.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] u99127 commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
u99127 commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r445098409



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -2566,17 +2613,27 @@ def convert_quantize(self, op):
         input_tensors = self.get_input_tensors(op)
         assert len(input_tensors) == 1, "input tensors length should be 1"
         input_tensor = input_tensors[0]
+        input_tensor_type_str = self.get_tensor_type_str(input_tensor.tensor.Type())
         in_expr = self.get_expr(input_tensor.tensor_idx)
 
         output_tensors = self.get_output_tensors(op)
         assert len(output_tensors) == 1, "output tensors length should be 1"
         output_tensor = output_tensors[0]
+        output_tensor_type_str = self.get_tensor_type_str(output_tensor.tensor.Type())
 
         # The output must be quantized
         assert output_tensor.qnn_params
-        # Quantize the input
-        out = self.quantize(in_expr, output_tensor)
 
+        # TFLite Quantize op can also act as Requantize op
+        if input_tensor_type_str == "float32":
+            out = self.quantize(in_expr, output_tensor)
+        else:
+            out = _qnn.op.requantize(in_expr,
+                                     input_scale=input_tensor.qnn_params['scale'],
+                                     input_zero_point=input_tensor.qnn_params['zero_point'],
+                                     output_scale=output_tensor.qnn_params['scale'],
+                                     output_zero_point=output_tensor.qnn_params['zero_point'],
+                                     out_dtype=output_tensor_type_str)
         return out

Review comment:
       This to me looks like it can go in by it's own right as a separate PR but this needs a unit test change in tflite/test_forward.py . 

##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -243,10 +243,46 @@ def get_tensors(self, tensors_idx_list):
             qnn_params = None
             tflite_qnn_params = tensor.Quantization()
             if tflite_qnn_params is not None:
-                scale = float(tflite_qnn_params.ScaleAsNumpy())
-                zero_point = int(tflite_qnn_params.ZeroPointAsNumpy())
+                # Params might be per-tensor or per-axis quantized. For per-tensor, scale and zero
+                # points are scalar. For per-axis, scale and zero points are tensors. But as per
+                # TFLite quantization spec, the restrictions on ops suggest that for per-axis, even
+                # if zero point is a tensor - all the zero points are identical.  More infomration
+                # here - https://www.tensorflow.org/lite/performance/quantization_spec
+
+                tflite_scale = tflite_qnn_params.ScaleAsNumpy()
+                tflite_zero_point = tflite_qnn_params.ZeroPointAsNumpy()
+                is_qnn_params_valid = True
+
+                # Handle Per-axis and per-tensor cases
+                if isinstance(tflite_scale, np.ndarray):
+                    assert isinstance(tflite_zero_point, np.ndarray)
+
+                    # Tensor - Per-axis quantization
+                    if tflite_scale.shape != (1,) and tflite_zero_point.shape != (1,):
+                        scale = tflite_scale
+                        # Ensure that all zero points are identical
+                        zero_point = tflite_zero_point
+                        assert all(x == zero_point[0] for x in zero_point)

Review comment:
       Minor Nit : Can we use an error here instead of an assert to show us clearly the change that has happened ? It also means we can provide some sensible diagnostic ?

##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -243,10 +243,46 @@ def get_tensors(self, tensors_idx_list):
             qnn_params = None
             tflite_qnn_params = tensor.Quantization()
             if tflite_qnn_params is not None:
-                scale = float(tflite_qnn_params.ScaleAsNumpy())
-                zero_point = int(tflite_qnn_params.ZeroPointAsNumpy())
+                # Params might be per-tensor or per-axis quantized. For per-tensor, scale and zero
+                # points are scalar. For per-axis, scale and zero points are tensors. But as per
+                # TFLite quantization spec, the restrictions on ops suggest that for per-axis, even
+                # if zero point is a tensor - all the zero points are identical.  More infomration
+                # here - https://www.tensorflow.org/lite/performance/quantization_spec

Review comment:
       To be clear, we are interpreting this from the fact that Conv2d and Depthwise_conv2d have a zero_point of 0 listed in their restriction even though they have per-axis quantization. 
   
   I would make the comment more explicit . 
   
   For per-axis or per-channel quantization the scale and zero points for the weights are tensors (?) 
   
   
   
   

##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -262,21 +298,25 @@ def get_tensor_value(self, tensor_wrapper):
         except ImportError:
             raise ImportError("The tflite package must be installed")
 
+        data = tensor_wrapper.buffer.DataAsNumpy()
+        shape = tensor_wrapper.tensor.ShapeAsNumpy()
+
+        # Set shape to 1 if the data is a scalar type
+        if data.shape == (1,) and isinstance(shape, int) and shape == 0:

Review comment:
       I'm scratching my head at this condition with shape. Can you elaborate more as to why we need it ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#issuecomment-653098040


   @u99127 can you please take a look again?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] siju-samuel merged pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
siju-samuel merged pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#issuecomment-648976903


   Opening the PR for review. @siju-samuel @masahi @u99127  @tqchen 
   
   The CI fails because of a missing package - tensorflow_hub 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r448586450



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -263,21 +305,29 @@ def get_tensor_value(self, tensor_wrapper):
         except ImportError:
             raise ImportError("The tflite package must be installed")
 
+        # Read the data from the buffer. Also extract the shape.
+        # The shape is used later to reshape the data.
+        data = tensor_wrapper.buffer.DataAsNumpy()
+        shape = tensor_wrapper.tensor.ShapeAsNumpy()
+
+        # When TFLite buffer is of size 1 (scalar), then TFLite tensor shape is set to 0.
+        # Therefore, we set the shape to 1 for numpy reshape to work.  Set shape to 1 if the data is
+        # a scalar type
+        if data.size == 1 and isinstance(shape, int) and shape == 0:
+            shape = (1,)
+
+        if tensor_wrapper.tensor.Type() == TensorType.INT8:

Review comment:
       For the first comment, thanks, let me take a look.
   
   For the second suggestion for has_same_qnn_params, I think we do not need that. For all the ops where we have to check params are same, they have scalar scale and zero point. This is because per-axis quantization is limited to weights, and thus limited to conv2d and dense op where we do not need this check.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] d-smirnov commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
d-smirnov commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r448572493



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -263,21 +305,29 @@ def get_tensor_value(self, tensor_wrapper):
         except ImportError:
             raise ImportError("The tflite package must be installed")
 
+        # Read the data from the buffer. Also extract the shape.
+        # The shape is used later to reshape the data.
+        data = tensor_wrapper.buffer.DataAsNumpy()
+        shape = tensor_wrapper.tensor.ShapeAsNumpy()
+
+        # When TFLite buffer is of size 1 (scalar), then TFLite tensor shape is set to 0.
+        # Therefore, we set the shape to 1 for numpy reshape to work.  Set shape to 1 if the data is
+        # a scalar type
+        if data.size == 1 and isinstance(shape, int) and shape == 0:
+            shape = (1,)
+
+        if tensor_wrapper.tensor.Type() == TensorType.INT8:

Review comment:
       Please consider following change as well:
   ```
       def has_same_qnn_params(self, lhs_tensor, rhs_tensor):
           lhs_scale = lhs_tensor.qnn_params['scale']
           rhs_scale = rhs_tensor.qnn_params['scale']
           lhs_zero_point = lhs_tensor.qnn_params['zero_point']
           rhs_zero_point = rhs_tensor.qnn_params['zero_point']
           return np.allclose( lhs_scale.data.asnumpy(), rhs_scale.data.asnumpy(), rtol=1e-5, atol=1e-5 ) and \
                  np.allclose( lhs_zero_point.data.asnumpy(), rhs_zero_point.data.asnumpy(), rtol=1e-5, atol=1e-5 )
   
   
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#discussion_r445326083



##########
File path: python/tvm/relay/frontend/tflite.py
##########
@@ -1605,7 +1645,7 @@ def convert_fully_connected(self, op):
 
         # weight tensor type should be UINT8 (quantization) or FLOAT32
         weight_tensor_type = weight_tensor.tensor.Type()
-        assert weight_tensor_type in (TensorType.UINT8, TensorType.FLOAT32)
+        assert weight_tensor_type in (TensorType.INT8, TensorType.UINT8, TensorType.FLOAT32)

Review comment:
       Good catch!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] siju-samuel commented on pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
siju-samuel commented on pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#issuecomment-652468515


   LGTM, Any more changes pending?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] siju-samuel commented on pull request #5848: [TFLite] QNN support for TFLite 2.1.0 quantized models

Posted by GitBox <gi...@apache.org>.
siju-samuel commented on pull request #5848:
URL: https://github.com/apache/incubator-tvm/pull/5848#issuecomment-653280854


   Thanks @anijain2305 @u99127 @tqchen. This is now merged.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org