[GitHub] [tvm] lhutton1 commented on pull request #13212: [TVMC][microNPU] tvmc option for printing which operators are offloaded to Ethos-U

[GitBox <gi...@apache.org>]

lhutton1 commented on PR #13212:
URL: https://github.com/apache/tvm/pull/13212#issuecomment-1312097771

   > > The option for printing the operators currently seems very specific to the NPU, I'm wondering if we would see more benefit adding this as a generic option within TVMC without too many changes? Not only would it benefit other targets, it would make the option more robust and easier to find from a user POV. Its currently possible to save the partitioned graph in TVMC using `--dump-code="relay"`, perhaps `print_operators_offloading` could be called at a similar point (given a command line argument such as `--dump-offloads`) rather than from within the NPU specific code, WDYT?
   > 
   > Hi @lhutton1,
   > 
   > Do you propose to implement this function for all the targets? Or just add a general compiler option leaving the implementation currently only in the ethos-u backend? Right now, this function is specific to ethos-u and is handled in the ethos-u backend. As far as I'm concerned it won't be a problem to implement the function for all targets, but of course I could be wrong.
   > 
   > Here is an example how the output would look like if model is compiled for the target "llvm":
   > 
   > ```
   >    'generic    <-   %0 = qnn.conv2d(%tfl.quantize, %v_param_1, ...'
   >    'generic    <-   %1 = nn.bias_add(%0, %v_param_2, axis=3);'
   >    'generic    <-   %2 = qnn.requantize(%1, meta[relay.Constant]...'
   > ```
   > 
   > And for targets "ethos-u,cmsis-nn,c"
   > 
   > ```
   >     'ethos-u    <- ethos-u.qnn_conv2d'
   >     'ethos-u    <-        %204 = qnn.conv2d(%203, %v_param_105, -128, 0, 0.0235294f, ...'
   >     'ethos-u    <-        %205 = nn.bias_add(%204, %v_param_106, axis=3);'
   >     'ethos-u    <-        %206 = qnn.requantize(%205, meta[relay.Constant][105], 0, ...'
   >     'ethos-u    <- ethos-u.reshape'
   >     'ethos-u    <-        %207 = reshape(%206, newshape=[1, 1001]);'
   >     'cmsis-nn   <- cmsis-nn.qnn_softmax'
   >     'cmsis-nn   <-        %208 = qnn.dequantize(%207, 0.0775722f, -61);'
   >     'cmsis-nn   <-        %209 = nn.softmax(%208);'
   >     'cmsis-nn   <-        qnn.quantize(%209, 0.00390625f, -128, out_dtype="int8")'
   > ```
   
   Thanks for the explanation @arina-grovety, yes I was thinking other backends like CMSIS-NN could make use of the same approach since the `AnalyzeOperationsDistribution` already seems quite generic. Where possible we can add the composite function names and if they are not found we can fallback to just printing the Relay - exactly as you described. If another backend has a different method of offloading operations this could simply be added to the pass in the future as and when needed.
   
   I see @ashutosh-arm's point that this feature intersects with the work in https://github.com/apache/tvm/issues/13116, perhaps it would be useful to have a discussion with the authors to align on expectations. In the future it would be great to see the `ethos-u.qnn_conv2d` annotations relate back to the input graph format (e.g. in TFLite: CONV2D) to make it easy for the user to relate their compiled operations to their original graph, but this seems a bit involved for now.
   
   > 1. `print_operators_offloading` would print the mapping of MicroNPU operators to the original Relay operators. Each target that uses partitioning has its own ways of defining these mappings. Some of them make use of `MergeCompilerRegions` that clubs multiple operators into a single partitioned function. Would it be possible to support additional targets given the knob name implies generic support in TVM?
   
   @ashutosh-arm, I think this is okay as operators will still be wrapped in their respective 'composite' function where the "ethos-u.qnn_conv2d" name is stored. Supporting other methods of operator offloading such as https://github.com/apache/tvm/blob/main/python/tvm/relay/op/contrib/ethosn.py#L451 I feel are out of scope for this work for the time being. I agree that we should get the community opinion on this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org