[GitHub] [tvm] rkimball commented on pull request #7428: Add pass to annotate ops with on_device for non-BYOC heterogeneous

[GitBox <gi...@apache.org>]

rkimball commented on pull request #7428:
URL: https://github.com/apache/tvm/pull/7428#issuecomment-782219803


   @comaniac and @manupa-arm to your questions
   This PR is to provide a simple demo of heterogeneous execution on CPU and Vulkan which are both tvm internal compilers.
   
   * I started with `compiler_begin` and `compiler_end` and was able to annotate my simple example model. These attributes worked great until up until calling the compilers. In my use case I need to compile code to use the CPU and Vulkan backends, both of which are built into tvm. With the `compiler_begin/end` annotation it really looks like it only works with external compilers, with no way to use tvm's built in compilers for the second device. For a quick demo I used the older `on_device` annotation which does directly work with both backends built into tvm.
   * There is a pass to annotate nodes, `AnnotateDevicePlacement` in `src/relay/transforms/annotate_device_placement.cc`. The pass simply call a callback for each CallNode and then annotates the nodes with the returned `on_device`. This pass is sufficient for the demo I had put together. What the user specifically wants to do is still under investigation.
   * As a more long-term goal I agree with you that consolidating the two heterogeneous approaches into a single approach is desired and hope to do that. AnnotateTarget looks to be complete. I though about wrapping CPU and GPU as "external" compilers but that seems like a more roundabout solution to the problem, better would be to unify the "external" and "internal" compilers so that either could be used with the `compile_begin/end` annotation.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org