You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/02/17 07:26:06 UTC

[GitHub] [incubator-tvm] jwfromm opened a new pull request #4899: [Relay][Frontend][Keras] NHWC import support.

jwfromm opened a new pull request #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899
 
 
   Currently the Keras importer can only output models with NCHW layout. This PR adds a layout argument to the Keras importer that allows NHWC to be used instead. This is useful for platforms that may have better NHWC than NCHW performance such as the Raspberry Pi.
   
   A few notes:
   * Xception is not being tested in NHWC mode as I found that it was causing a strange cuda error when targeting GPU. On CPU, it works fine however so I suspect this is an issue with the nhwc separable convolutions in Topi rather than the importer.
   * I made a few changes to the test script to allow eager execution (and TF2.0) to work.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tqchen commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-587611716
 
 
   Merging this for now as this PR is self contained. It would be great if we can have a followup PR that updates the frontend to use ConvertLayout from NHWC -> NCHW so we won't have to support two path in the frontend

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] jwfromm commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
jwfromm commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-587153715
 
 
   @anijain2305, I think you're right that for most cases ConvertLayout will achieve something similar. However, there are some benefits to having layers directly parsed as NHWC. I actually did this PR because I'm working on a project that uses some funky custom Keras layers and it's very useful to parse them directly in NHWC.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#discussion_r380299492
 
 

 ##########
 File path: python/tvm/relay/frontend/keras.py
 ##########
 @@ -234,18 +234,29 @@ def _convert_dense(inexpr, keras_layer, etab):
 
 def _convert_convolution(inexpr, keras_layer, etab):
     _check_data_format(keras_layer)
+    if etab.data_layout == 'NHWC':
+        kernel_layout = 'HWIO'
+    else:
+        kernel_layout = 'OIHW'
     is_deconv = type(keras_layer).__name__ == 'Conv2DTranspose'
     is_depthconv = type(keras_layer).__name__ == 'DepthwiseConv2D'
     weightList = keras_layer.get_weights()
+    weight = weightList[0]
     if is_deconv:
-        kernel_h, kernel_w, n_filters, in_channels = weightList[0].shape
-        weight = weightList[0].transpose([3, 2, 0, 1])
+        kernel_h, kernel_w, n_filters, in_channels = weight.shape
+        if kernel_layout == 'OIHW':
+            weight = weight.transpose([3, 2, 0, 1])
     elif is_depthconv:
-        kernel_h, kernel_w, in_channels, depth_mult = weightList[0].shape
-        weight = weightList[0].transpose([2, 3, 0, 1])
+        kernel_h, kernel_w, in_channels, depth_mult = weight.shape
+        if kernel_layout == 'OIHW':
+            weight = weight.transpose([2, 3, 0, 1])
+        else:
+            kernel_layout = "HWOI"
 
 Review comment:
   This looks confusion at the first glance. It might be better to put all `kernel_layout` determination related logic together.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] anijain2305 commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-587173870
 
 
   I see. Yes, I meant that we should retain native framework format (NHWC in this case). And use ConvertLayout to change the layout to desired layout (passed on by relay.from_keras API).
   
   @jwfromm Yes, in my opinion, that is the cleanest way forward. Same thing needs to happen for TF framework. I can take care of that. Will appreciate if you can take care of Keras.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] jwfromm commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
jwfromm commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-587118058
 
 
   @comaniac I think I've fixed all the style issues you pointed out.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tqchen commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-587154754
 
 
   Given that the model is in native NHWC, perhaps it is better to do it in another way around, ingest as NHWC as in this PR and then call ConvertLayout

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] masahi edited a comment on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
masahi edited a comment on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-586856971
 
 
   I'm not familiar with keras, but I wonder why keras use NCHW while tensorflow uses NHWC? (I'm not familiar with tf either)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tqchen edited a comment on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-587611716
 
 
   Merging this for now as this PR is self contained. It would be great if we can have a followup PR that updates the frontend to use ConvertLayout from NHWC -> NCHW so we won't have to support two path in the frontend
   
   Thanks @anijain2305 @jwfromm @comaniac 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] jwfromm commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
jwfromm commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-587170389
 
 
   @tqchen, you're right that would be a more direct way to apply ConvertLayout. If everyone else would prefer it, I could change the Keras frontend to only output in NHWC and then we would rely on ConvertLayout to get models into NCHW.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#discussion_r380300117
 
 

 ##########
 File path: python/tvm/relay/frontend/keras.py
 ##########
 @@ -274,8 +287,13 @@ def _convert_convolution(inexpr, keras_layer, etab):
         if pad_t == pad_b and pad_l == pad_r:
             params['padding'] = (pad_t, pad_l)
         else:
-            inexpr = _op.nn.pad(data=inexpr, pad_width=(
-                (0, 0), (0, 0), (pad_t, pad_b), (pad_l, pad_r)))
+            if etab.data_layout == 'NCHW':
 
 Review comment:
   ```python
   if pad_t ...:
     ...
   elif etab.data_layout == 'NCHW':
     ...
   else:
     ...
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] jwfromm commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
jwfromm commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-587108875
 
 
   @masahi, Keras does use NHWC by default just like TensorFlow. However, because TVM is better optimized for NCHW, the Keras frontend previously converted all layers from NHWC to NCHW. This option allows us to keep the layers in their original format.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] masahi edited a comment on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
masahi edited a comment on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-586856971
 
 
   I'm not familiar with keras, but I wonder why keras use NCHW while tensorflow uses NHWC? (I'm not familiar with tf either). I heard that since TF 2.0, keras became their official frontend or something like that.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] jwfromm commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
jwfromm commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-586852298
 
 
   @masahi, @soiferj  Can you take a look at this PR?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tqchen commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-587224358
 
 
   That sounds good. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#discussion_r380300684
 
 

 ##########
 File path: python/tvm/relay/frontend/keras.py
 ##########
 @@ -322,26 +353,39 @@ def _convert_separable_convolution(inexpr, keras_layer, etab):
         if pad_t == pad_b and pad_l == pad_r:
             params0['padding'] = (pad_t, pad_l)
         else:
-            inexpr = _op.nn.pad(data=inexpr, pad_width=(
-                (0, 0), (0, 0), (pad_t, pad_b), (pad_l, pad_r)))
+            if etab.data_layout == 'NCHW':
 
 Review comment:
   ditto

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] masahi commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
masahi commented on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-586856971
 
 
   I'm not familiar with keras, but I wonder why keras use NCHW while tensorflow uses NHWC?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tqchen edited a comment on issue #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on issue #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#issuecomment-587154754
 
 
   Given that the model is in native NHWC, perhaps it is better to do it in another way around(note that the old approach converts to NCHW on the fly), ingest as NHWC as in this PR and then call ConvertLayout

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] comaniac commented on a change in pull request #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899#discussion_r380301754
 
 

 ##########
 File path: python/tvm/relay/frontend/keras.py
 ##########
 @@ -751,6 +816,10 @@ def from_keras(model, shape=None):
     shape: dict of str to int list/tuple
         Input shapes of the model, optional
 
+    layout: str
 
 Review comment:
   Should we mention like "default layout is NCHW"?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-tvm] tqchen merged pull request #4899: [Relay][Frontend][Keras] NHWC import support.

Posted by GitBox <gi...@apache.org>.
tqchen merged pull request #4899: [Relay][Frontend][Keras] NHWC import support.
URL: https://github.com/apache/incubator-tvm/pull/4899
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services