You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/09/10 12:37:51 UTC

[GitHub] [incubator-tvm] Beya2019 opened a new pull request #6443: [RELAY][OP] roi_align operator alter layout

Beya2019 opened a new pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443


   RFC: #4335
   https://discuss.tvm.ai/t/layout-conversion-pass/4009
   
   add aoi_align operator(in maskrcnn) convert_op_layout and related test case in test_pass_convert_op_layout.py .
   
   Would you please have a look at this @yzhliu  @vinx13 @anijain2305 @tqchen 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] xqdan commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
xqdan commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r487824780



##########
File path: tests/python/relay/test_op_level5.py
##########
@@ -1142,14 +1159,14 @@ def verify_grid_sample(data_shape, grid_shape):
 
 
 if __name__ == "__main__":
-    test_resize_infer_type()
-    test_resize()
-    test_resize3d_infer_type()
-    test_resize3d()
-    test_crop_and_resize()
-    test_multibox_prior()
-    test_multibox_transform_loc()
-    test_get_valid_counts()
+    # test_resize_infer_type()

Review comment:
       remove these




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r487234456



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @Beya2019 ,
   We understand your usecase with the special target. But, when we put add changes in TVM, we want to ensure that we don't break things for x86, arm and gpus. Now, what @kevinthesun and I are saying is that if you try compiling a graph for x86(or gpu) with this PR, you should see compilation failure. Our reasoning is that the ConvertLayout will introduce roi_align with NHWC layout, and then InferType will be called, this will cause failure. Can you please try this and confirm?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486813081



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun 
   
   "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486679507



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Do we really need another layout for this op, since layout won't affect performance much for it?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r487234456



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @Beya2019 ,
   We understand your usecase with the special target. But, when we put add changes in TVM, we want to ensure that we don't break things for x86, arm and gpus. Now, what @kevinthesun and I are saying is that if you try compiling a graph for x86(or gpu) with this PR, you should see compilation failure. Our reasoning is that the ConvertLayout will introduce roi_align with NHWC layout, and then InferType will be called, this will cause failure. Can you please try this and confirm?

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @Beya2019 ,
   We understand your usecase with the special target. But, when we put add changes in TVM, we want to ensure that we don't break things for x86, arm and gpus. Now, what @kevinthesun and I are saying is that if you try compiling a graph for x86(or gpu) with this PR, you should see compilation failure. Our reasoning is that the ConvertLayout will introduce roi_align with NHWC layout, and then InferType will be called, this will cause failure. Can you please try this and confirm?

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @Beya2019 ,
   We understand your usecase with the special target. But, when we put add changes in TVM, we want to ensure that we don't break things for x86, arm and gpus. Now, what @kevinthesun and I are saying is that if you try compiling a graph for x86(or gpu) with this PR, you should see compilation failure. Our reasoning is that the ConvertLayout will introduce roi_align with NHWC layout, and then InferType will be called, this will cause failure. Can you please try this and confirm?

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @Beya2019 ,
   We understand your usecase with the special target. But, when we put add changes in TVM, we want to ensure that we don't break things for x86, arm and gpus. Now, what @kevinthesun and I are saying is that if you try compiling a graph for x86(or gpu) with this PR, you should see compilation failure. Our reasoning is that the ConvertLayout will introduce roi_align with NHWC layout, and then InferType will be called, this will cause failure. Can you please try this and confirm?

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @Beya2019 ,
   We understand your usecase with the special target. But, when we put add changes in TVM, we want to ensure that we don't break things for x86, arm and gpus. Now, what @kevinthesun and I are saying is that if you try compiling a graph for x86(or gpu) with this PR, you should see compilation failure. Our reasoning is that the ConvertLayout will introduce roi_align with NHWC layout, and then InferType will be called, this will cause failure. Can you please try this and confirm?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 edited a comment on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 edited a comment on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-694165290


   Hi @kevinthesun:
      I have fixed it. Please review. 
      In addition, I find #6449 has merged. But you do not enable the  test_detection_models() test case in tests/python/frontend/pytorch/test_object_detection.py, and must be tested manually. So the tvm ci system cannot test this case automatically. Please note it.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-694391738


   > Hi @kevinthesun:
   > I have fixed it. Please review.
   > In addition, I find #6449 has merged. But you do not enable the test_detection_models() test case in tests/python/frontend/pytorch/test_object_detection.py, and must be tested manually. So the tvm ci system cannot test this case automatically. Please note it.
   
   It is tested by pytest.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r489716429



##########
File path: src/relay/op/vision/rcnn_op.cc
##########
@@ -43,14 +45,43 @@ bool ROIAlignRel(const Array<Type>& types, int num_inputs, const Attrs& attrs,
   CHECK(roi_align_attrs);
   CHECK_EQ(dshape.size(), 4) << "Input data should be 4-D.";
   CHECK_EQ(rshape.size(), 2) << "Input rois should be 2-D.";
-  CHECK_EQ(roi_align_attrs->layout, "NCHW") << "ROI Align only supports NCHW layout";
   // assign output type
-  std::vector<IndexExpr> oshape(
-      {rshape[0], dshape[1], roi_align_attrs->pooled_size[0], roi_align_attrs->pooled_size[1]});
+  std::vector<IndexExpr> oshape;
+  if (roi_align_attrs->layout == "NCHW") {
+    oshape = {rshape[0], dshape[1], roi_align_attrs->pooled_size[0],
+              roi_align_attrs->pooled_size[1]};
+  } else {
+    CHECK_EQ(roi_align_attrs->layout, "NHWC") << "Unexpected ROI Align layout";
+    oshape = {rshape[0], roi_align_attrs->pooled_size[0], roi_align_attrs->pooled_size[1],
+              dshape[3]};
+  }
+
   reporter->Assign(types[2], TensorType(oshape, data->dtype));
   return true;
 }
 
+template <typename T>
+Array<Array<Layout> > ROIAlignInferCorrectLayout(const Attrs& attrs,

Review comment:
       The issue can be reproduced by applying this patch to https://github.com/apache/incubator-tvm/pull/6449 and run ```test_detection_models()``` in ```tests/python/frontend/pytorch/test_object_detection.py```.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r487234456



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @Beya2019 ,
   We understand your usecase with the special target. But, when we put add changes in TVM, we want to ensure that we don't break things for x86, arm and gpus. Now, what @kevinthesun and I are saying is that if you try compiling a graph for x86(or gpu) with this PR, you should see compilation failure. Our reasoning is that the ConvertLayout will introduce roi_align with NHWC layout, and then InferType will be called, this will cause failure. Can you please try this and confirm?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-694165290


   Hi kevinthesun :
      I have fixed it. Please review. 
      In addition, I find #6449 has merged. But you do not enable the  test_detection_models() test case in tests/python/frontend/pytorch/test_object_detection.py, and must be tested manually. So the tvm ci system cannot test this case automatically. Please note it.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486813081



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun 
   
   "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486803979



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level NHWC roi_align will fail during type inference since we do check NCHW layout: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align even on relay level, we might want to remove this check and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-692521267


   Hi, @kevinthesun
   
   I have move the check for NCHW from relay type infer into op strategy for corresponding targets. eg:
   ```
       layout = attrs.layout
       if layout == "NCHW":
           strategy.add_implementation(
               wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
               wrap_topi_schedule(topi.generic.schedule_roi_align),
               name="roi_align_nchw.generic",
           )
       elif layout == "NHWC":
           strategy.add_implementation(
               wrap_compute_roi_align(topi.vision.rcnn.roi_align_nhwc),
               wrap_topi_schedule(topi.generic.schedule_roi_align),
               name="roi_align_nhwc.generic",
           )
   ```
   
   we depend on the attrs.layout to call different compute and schedule. and we realize the nhwc layout basic compute refered to nchw layout compute to ensure the the basic process is passable.
   
   In additon, What I understand is: different targets support different layouts, and the user chooses the high-performance layout supported by his target. The right to choose which layout is open to users, users can use transform.ConvertLayout(desired_layouts) to convert between NHWC and NCHW layouts. And the nhwc layout compute we added is to ensure that the conversion will not cause compilation errors. In the feature, if other users have the performance demand for GPU or other targets, he can add the corresponding schedule to improve performance. The current implementation only provides a general method.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486803979



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level NHWC roi_align will fail during type inference since we do check NCHW layout: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align even on relay level, we might want to remove this check and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this contraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this constraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Today we don't check roi_align layout in op strategy since we do the check in relay infer type level. If we want to allow NHWC in relay level, we need to remove that constraint in ROIAlignRel and move it here. Silently selecting nchw implementation and fail in later stage(even runtime) is not great.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level NHWC roi_align will fail during type inference since we do check NCHW layout: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align even on relay level, we might want to remove this check and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this contraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this constraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Today we don't check roi_align layout in op strategy since we do the check in relay infer type level. If we want to allow NHWC in relay level, we need to remove that constraint in ROIAlignRel and move it here. Silently selecting nchw implementation and fail in later stage(even runtime) is not great.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level NHWC roi_align will fail during type inference since we do check NCHW layout: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align even on relay level, we might want to remove this check and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this contraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this constraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Today we don't check roi_align layout in op strategy since we do the check in relay infer type level. If we want to allow NHWC in relay level, we need to remove that constraint in ROIAlignRel and move it here. Silently selecting nchw implementation and fail in later stage(even runtime) is not great.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level NHWC roi_align will fail during type inference since we do check NCHW layout: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align even on relay level, we might want to remove this check and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this contraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this constraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Today we don't check roi_align layout in op strategy since we do the check in relay infer type level. If we want to allow NHWC in relay level, we need to remove that constraint in ROIAlignRel and move it here. Silently selecting nchw implementation and fail in later stage(even runtime) is not great.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level NHWC roi_align will fail during type inference since we do check NCHW layout: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align even on relay level, we might want to remove this check and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this contraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this constraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Today we don't check roi_align layout in op strategy since we do the check in relay infer type level. If we want to allow NHWC in relay level, we need to remove that constraint in ROIAlignRel and move it here. Silently selecting nchw implementation and fail in later stage(even runtime) is not great.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level NHWC roi_align will fail during type inference since we do check NCHW layout: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align even on relay level, we might want to remove this check and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this contraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this constraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Today we don't check roi_align layout in op strategy since we do the check in relay infer type level. If we want to allow NHWC in relay level, we need to remove that constraint in ROIAlignRel and move it here. Silently selecting nchw implementation and fail in later stage(even runtime) is not great.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level NHWC roi_align will fail during type inference since we do check NCHW layout: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align even on relay level, we might want to remove this check and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this contraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this constraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level NHWC roi_align will fail during type inference since we do check NCHW layout: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align even on relay level, we might want to remove this check and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this contraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this constraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Today we don't check roi_align layout in op strategy since we do the check in relay infer type level. If we want to allow NHWC in relay level, we need to remove that constraint in ROIAlignRel and move it here. Silently selecting nchw implementation and fail in later stage(even runtime) is not great.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level NHWC roi_align will fail during type inference since we do check NCHW layout: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align even on relay level, we might want to remove this check and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this contraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this constraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Today we don't check roi_align layout in op strategy since we do the check in relay infer type level. If we want to allow NHWC in relay level, we need to remove that constraint in ROIAlignRel and move it here. Silently selecting nchw implementation and fail in later stage(even runtime) is not great.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-690523605


   Is there a `NHWC` topi implementation for `roi_align`? If not, this will cause compilation failure for traditional TVM targets.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486734730



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @kevinthesun and @anijain2305,
   
    This NHWC layout topi implementation for roi_align is really not currently supported for traditional TVM targets. But in relay/frontend/mxnet.py the _mx_roi_align operator is fixed to NCHW(new_attrs["layout"] = "NCHW"),  that is only the nchw topi implemention process is called normally. For the special target which is only support nhwc layout, we can add extra code in mxnet.py(or other frontend py) as follows(RFC:https://tvm.apache.org/docs/dev/convert_layout.html#usage):
   ```
   desired_layouts = {'vision.roi_align': ['NHWC', `default`]}
   
   # Convert the layout to NCHW
   # RemoveUnunsedFunctions is used to clean up the graph.
   seq = tvm.transform.Sequential([relay.transform.RemoveUnusedFunctions(),
                                   relay.transform.ConvertLayout(desired_layouts)])
   with tvm.transform.PassContext(opt_level=3):
       mod = seq(mod)
   ```
   So, it doesn't cause the compilation failure for the nhwc topi implementation process can not be called for traditional TVM targets. but it expanded the nhwc relay ir implement with their own backend code for the special target which is only support nhwc layout.
   
   
   Certainly, the nhwc layout topi implementation is to be realized sooner or later, I think it can be improved gradually in the future.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486679507



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Do we really need another layout for this op?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486803979



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level NHWC roi_align will fail during type inference since we do check NCHW layout: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align even on relay level, we might want to remove this check and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this contraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this constraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Today we don't check roi_align layout in op strategy since we do the check in relay infer type level. If we want to allow NHWC in relay level, we need to remove that constraint in ROIAlignRel and move it here. Silently selecting nchw implementation and fail in later stage(even runtime) is not great.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-693132640


   @kevinthesun :I got it. According to your suggestion, I modified the code. Please review again, does it meets the demans? If you have any suggestions, please feedback in time? Thanks very much.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] xqdan commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
xqdan commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-692519854


   
   
   
   > @xqdan @Beya2019 I'm not quite sure about whether we really need a NHWC layout topi implementation for roi_align on general purpose hardware, since IMO this should not become any major concern for performance. What @anijain2305 and I are suggesting is that we move the check for NCHW from relay type infer into op strategy for corresponding targets which don't have NHWC implementation.
   > 
   > I agree with @xqdan that topi implementation doesn't need to bind with relay layout. As a result, can we have some data showing the performance difference between NCHW and NHWC layouts roi_align for CPU and a new topi implementation is necessary?
   
   Maybe just remove NHWC layouts roi_align, and make sure relay type infer check for different target, make it easy for @Beya2019 , what do you think? @kevinthesun 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-692007440


   Hi, @kevinthesun @anijain2305 
   
   According to your suggestion, i have added the topi compute and schedule of NHWC layout for x86, arm target. Meantime I add the test case in tests/python/relay/test_op_level5.py / tests/python/relay/test_pass_convert_op_layout.py / tests/python/relay/test_pass_convert_op_layout.py to confirm the correctness of calculation and ensure there is no compile failure due to the introduction of the ConvertLayout for roi_align NHWC layout and InferType is be called. Can you help me to have a review again and merge?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486679507



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Do we really need another layout for this op? Layout won't affect performance much for it?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486742150



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun and @anijain2305, In addition, I would like to add one more point.
   
   
   In our own target, this implementation method really affect our performance for it add many layout_transpose operator(Our convolution only supports nhwc, if roi_align only supports nchw, it means that many additional layout_transform operators need to be inserted into the network), I also think really need nhwc layout implementation for this op considering performance. And this submission is needed for layout convert support whether nhwc layout realized or not. The two mentioned function are not conflicting but complementary, I also hope the nhwc layout topi implementation as soon as possible.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-692922971


   @xqdan That's what I meant.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r487227873



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Today we don't check roi_align layout in op strategy since we do the check in relay infer type level. If we want to allow NHWC in relay level, we need to remove that constraint in ROIAlignRel and move it here. Silently selecting nchw implementation and fail in later stage(even runtime) is not great.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486813081



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486734730



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @kevinthesun and @anijain2305,
   
    This NHWC layout topi implementation for roi_align is really not currently supported for traditional TVM targets. But in relay/frontend/mxnet.py the _mx_roi_align operator is fixed to NCHW(new_attrs["layout"] = "NCHW"),  that is only the nchw topi implemention process is called normally. For the special target which is only support nhwc layout, we can add extra code in mxnet.py(or other frontend py) as follows(RFC:https://tvm.apache.org/docs/dev/convert_layout.html#usage):
   ```
   desired_layouts = {'vision.roi_align': ['NHWC', `default`]}
   
   # Convert the layout to NCHW
   # RemoveUnunsedFunctions is used to clean up the graph.
   seq = tvm.transform.Sequential([relay.transform.RemoveUnusedFunctions(),
                                   relay.transform.ConvertLayout(desired_layouts)])
   with tvm.transform.PassContext(opt_level=3):
       mod = seq(mod)
   ```
   So, it doesn't cause the compilation failure for the nhwc topi implementation process can not be called for traditional TVM targets. but it expanded the nhwc relay ir implement with their own backend code for the special target which is only support nhwc layout.
   
   
   Certainly, the nhwc layout topi implementation is to be realized sooner or later, I think it can be improved gradually in the future.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun and @anijain2305, In addition, I would like to add one more point.
   
   
   In our own target, this implementation method really affect our performance for it add many layout_transpose operator(Our convolution only supports nhwc, if roi_align only supports nchw, it means that many additional layout_transform operators need to be inserted into the network), I also think really need nhwc layout implementation for this op considering performance. And this submission is needed for layout convert support whether nhwc layout realized or not. The two mentioned function are not conflicting but complementary, I also hope the nhwc layout topi implementation as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun 
   
   "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486734730



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @kevinthesun and @anijain2305,
   
    This NHWC layout topi implementation for roi_align is really not currently supported for traditional TVM targets. But in relay/frontend/mxnet.py the _mx_roi_align operator is fixed to NCHW(new_attrs["layout"] = "NCHW"),  that is only the nchw topi implemention process is called normally. For the special target which is only support nhwc layout, we can add extra code in mxnet.py(or other frontend py) as follows(RFC:https://tvm.apache.org/docs/dev/convert_layout.html#usage):
   ```
   desired_layouts = {'vision.roi_align': ['NHWC', `default`]}
   
   # Convert the layout to NCHW
   # RemoveUnunsedFunctions is used to clean up the graph.
   seq = tvm.transform.Sequential([relay.transform.RemoveUnusedFunctions(),
                                   relay.transform.ConvertLayout(desired_layouts)])
   with tvm.transform.PassContext(opt_level=3):
       mod = seq(mod)
   ```
   So, it doesn't cause the compilation failure for the nhwc topi implementation process can not be called for traditional TVM targets. but it expanded the nhwc relay ir implement with their own backend code for the special target which is only support nhwc layout.
   
   
   Certainly, the nhwc layout topi implementation is to be realized sooner or later, I think it can be improved gradually in the future.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun and @anijain2305, In addition, I would like to add one more point.
   
   
   In our own target, this implementation method really affect our performance for it add many layout_transpose operator(Our convolution only supports nhwc, if roi_align only supports nchw, it means that many additional layout_transform operators need to be inserted into the network), I also think really need nhwc layout implementation for this op considering performance. And this submission is needed for layout convert support whether nhwc layout realized or not. The two mentioned function are not conflicting but complementary, I also hope the nhwc layout topi implementation as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun 
   
   "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun 
   
   "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun 
   
   "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun 
   
   "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun 
   
   "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun 
   
   "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun 
   
   "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       @kevinthesun 
   
   "we need remove this constraint for relay type inference and move it to op strategy for corresponding targets"
   
   For the issue, there already exists the constraint as:
   
   `python/tvm/relay/op/strategy/cuda.py`
   ```
   @roi_align_strategy.register(["cuda", "gpu"])
   def roi_align_strategy_cuda(attrs, inputs, out_type, target):
       """roi_align cuda strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.vision.rcnn.roi_align_nchw),
                                   wrap_topi_schedule(topi.cuda.schedule_roi_align),
                                   name="roi_align_nchw.cuda")
       return strategy
   ```
   
   and `python/tvm/relay/op/strategy/x86.py`
   
   ```
   @roi_align_strategy.register("cpu")
   def roi_align_strategy_cpu(attrs, inputs, out_type, target):
       """roi_align x86 strategy"""
       strategy = _op.OpStrategy()
       strategy.add_implementation(wrap_compute_roi_align(topi.x86.roi_align_nchw),
                                   wrap_topi_schedule(topi.generic.schedule_roi_align),
                                   name="roi_align.x86")
       return strategy
   ```
   
   In future, we just need to realize the topi.vision.rcnn.roi_align_nhwc and add the constraint(call topi.vision.rcnn.roi_align_nhwc) to the special target. And I will realize this job as soon as possible.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] Beya2019 edited a comment on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
Beya2019 edited a comment on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-692007440


   Hi, @kevinthesun @anijain2305 
   
   According to your suggestion, i have added the topi compute and schedule of NHWC layout for x86, arm target. Meantime I add the test case in tests/python/relay/test_op_level5.py / tests/python/topi/python/test_topi_vision.py / tests/python/relay/test_pass_convert_op_layout.py to confirm the correctness of calculation and ensure there is no compile failure due to the introduction of the ConvertLayout for roi_align NHWC layout and InferType is be called. Can you help me to have a review again and merge?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-690774753


   Yeah. As @anijain2305 mentioned, we only have ```roi_align_nchw``` in topi.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486803979



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this contraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r489701599



##########
File path: src/relay/op/vision/rcnn_op.cc
##########
@@ -43,14 +45,43 @@ bool ROIAlignRel(const Array<Type>& types, int num_inputs, const Attrs& attrs,
   CHECK(roi_align_attrs);
   CHECK_EQ(dshape.size(), 4) << "Input data should be 4-D.";
   CHECK_EQ(rshape.size(), 2) << "Input rois should be 2-D.";
-  CHECK_EQ(roi_align_attrs->layout, "NCHW") << "ROI Align only supports NCHW layout";
   // assign output type
-  std::vector<IndexExpr> oshape(
-      {rshape[0], dshape[1], roi_align_attrs->pooled_size[0], roi_align_attrs->pooled_size[1]});
+  std::vector<IndexExpr> oshape;
+  if (roi_align_attrs->layout == "NCHW") {
+    oshape = {rshape[0], dshape[1], roi_align_attrs->pooled_size[0],
+              roi_align_attrs->pooled_size[1]};
+  } else {
+    CHECK_EQ(roi_align_attrs->layout, "NHWC") << "Unexpected ROI Align layout";
+    oshape = {rshape[0], roi_align_attrs->pooled_size[0], roi_align_attrs->pooled_size[1],
+              dshape[3]};
+  }
+
   reporter->Assign(types[2], TensorType(oshape, data->dtype));
   return true;
 }
 
+template <typename T>
+Array<Array<Layout> > ROIAlignInferCorrectLayout(const Attrs& attrs,

Review comment:
       There are still issues for ```ROIAlignInferCorrectLayout```. I applied this patch and pytorch od model compilation will fail with seg fault.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-694992667


   Thanks @Beya2019 @anijain2305 @xqdan 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] anijain2305 commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
anijain2305 commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r487234456



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @Beya2019 ,
   We understand your usecase with the special target. But, when we put add changes in TVM, we want to ensure that we don't break things for x86, arm and gpus. Now, what @kevinthesun and I are saying is that if you try compiling a graph for x86(or gpu) with this PR, you should see compilation failure. Our reasoning is that the ConvertLayout will introduce roi_align with NHWC layout, and then InferType will be called, this will cause failure. Can you please try this and confirm?

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @Beya2019 ,
   We understand your usecase with the special target. But, when we put add changes in TVM, we want to ensure that we don't break things for x86, arm and gpus. Now, what @kevinthesun and I are saying is that if you try compiling a graph for x86(or gpu) with this PR, you should see compilation failure. Our reasoning is that the ConvertLayout will introduce roi_align with NHWC layout, and then InferType will be called, this will cause failure. Can you please try this and confirm?

##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       Hi @Beya2019 ,
   We understand your usecase with the special target. But, when we put add changes in TVM, we want to ensure that we don't break things for x86, arm and gpus. Now, what @kevinthesun and I are saying is that if you try compiling a graph for x86(or gpu) with this PR, you should see compilation failure. Our reasoning is that the ConvertLayout will introduce roi_align with NHWC layout, and then InferType will be called, this will cause failure. Can you please try this and confirm?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-692217439


   @xqdan @Beya2019 I'm not quite sure about whether we really need a NHWC layout topi implementation for roi_align on general purpose hardware, since IMO this should not become any major concern for performance. What @anijain2305 and I are suggesting is that we move the check for NCHW from relay type infer into op strategy for corresponding targets which don't have NHWC implementation. 
   
   I agree with @xqdan that topi implementation doesn't need to bind with relay layout. As a result, can we have some data showing the performance difference between NCHW and NHWC layouts roi_align for CPU and a new topi implementation is necessary?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r486803979



##########
File path: python/tvm/relay/op/vision/rcnn.py
##########
@@ -24,7 +24,7 @@ def roi_align(data, rois, pooled_size, spatial_scale, sample_ratio=-1, layout='N
     Parameters
     ----------
     data : relay.Expr
-        4-D tensor with shape [batch, channel, height, width]
+        4-D tensor with shape [batch, channel, height, width] or [batch, height, width, channel]

Review comment:
       OK. One thing to note is that on relay level we do check NCHW layout for roi_align: https://github.com/apache/incubator-tvm/blob/master/src/relay/op/vision/rcnn_op.cc#L46 If we would like to allow NHWC roi_align, we might want to remove this constraint for relay type inference and move it to op strategy for corresponding targets, since we don't have implementation for NHWC and usually layout for roi_align is not an issue in end to end inference for these general purpose targets.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-692926700


   @Beya2019 Thanks for adding nhwc implementation. However, this also adds maintenance burden if there is no obvious performance benefit. Recently we are adding dynamic shape support for some ops. If we just keep necessary implementations it would be easier to maintain and extend.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] xqdan edited a comment on pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
xqdan edited a comment on pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#issuecomment-692519854


   
   
   > @xqdan @Beya2019 I'm not quite sure about whether we really need a NHWC layout topi implementation for roi_align on general purpose hardware, since IMO this should not become any major concern for performance. What @anijain2305 and I are suggesting is that we move the check for NCHW from relay type infer into op strategy for corresponding targets which don't have NHWC implementation.
   > 
   > I agree with @xqdan that topi implementation doesn't need to bind with relay layout. As a result, can we have some data showing the performance difference between NCHW and NHWC layouts roi_align for CPU and a new topi implementation is necessary?
   
   Maybe just remove NHWC layouts roi_align cpu op, and make sure relay type infer check for different target, make it easy for @Beya2019 , what do you think? @kevinthesun 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun merged pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun merged pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-tvm] kevinthesun commented on a change in pull request #6443: [RELAY][OP] roi_align operator alter layout

Posted by GitBox <gi...@apache.org>.
kevinthesun commented on a change in pull request #6443:
URL: https://github.com/apache/incubator-tvm/pull/6443#discussion_r489701599



##########
File path: src/relay/op/vision/rcnn_op.cc
##########
@@ -43,14 +45,43 @@ bool ROIAlignRel(const Array<Type>& types, int num_inputs, const Attrs& attrs,
   CHECK(roi_align_attrs);
   CHECK_EQ(dshape.size(), 4) << "Input data should be 4-D.";
   CHECK_EQ(rshape.size(), 2) << "Input rois should be 2-D.";
-  CHECK_EQ(roi_align_attrs->layout, "NCHW") << "ROI Align only supports NCHW layout";
   // assign output type
-  std::vector<IndexExpr> oshape(
-      {rshape[0], dshape[1], roi_align_attrs->pooled_size[0], roi_align_attrs->pooled_size[1]});
+  std::vector<IndexExpr> oshape;
+  if (roi_align_attrs->layout == "NCHW") {
+    oshape = {rshape[0], dshape[1], roi_align_attrs->pooled_size[0],
+              roi_align_attrs->pooled_size[1]};
+  } else {
+    CHECK_EQ(roi_align_attrs->layout, "NHWC") << "Unexpected ROI Align layout";
+    oshape = {rshape[0], roi_align_attrs->pooled_size[0], roi_align_attrs->pooled_size[1],
+              dshape[3]};
+  }
+
   reporter->Assign(types[2], TensorType(oshape, data->dtype));
   return true;
 }
 
+template <typename T>
+Array<Array<Layout> > ROIAlignInferCorrectLayout(const Attrs& attrs,

Review comment:
       There are still issues for ```ROIAlignInferCorrectLayout```. I applied this patch and pytorch od model compilation will cause seg fault.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org