You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/09/29 01:27:03 UTC
[GitHub] [tvm-rfcs] mbs-octoml opened a new pull request #38: [RFC] Improved multi-target handling
mbs-octoml opened a new pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38
While working on unifying some device planning passes and cleaning up the TECompiler we noticed we have a lot of issues around how we transition from devices to Targets in the hetogeneous compilation flow. This pre-RFC is a stab at fixing those. There's a somewhat larger issue around bringing BYOC target name annotation and device planning closer together since they are very very similar but currently incompatible. However I don't understand the BYOC flow enough to launch into that.
I realize we normally start with a pre-RFC discussion on discord, but would like to experiment with having that conversation directly on GitHub. That way I can update the RFC text to account for discussion as we go, capturing where we have consensus and capturing alternatives directly in the text. Humor me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml commented on pull request #38: [RFC] Unified device/target/memory scope planning
Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#issuecomment-1064590913
Closing as obsolete, since most of this is either already done or has been subsumed by the Collage proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml commented on a change in pull request #38: [RFC] Unified device/target/memory scope planning
Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on a change in pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#discussion_r734033010
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly
+ devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer
+ uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an
+ arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators.
+ Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system.
+3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for
+ primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the
+ overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to
+ deal with these different conventions.
+4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations
+ should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied
+ `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that
+ `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets,
+ it is particularly important to ensure there's a single source of truth for available `Target`s.
+5. TVM also supports a 'BYOC' extension mechanism. This allows `"target.<target name>"` annotations to be placed on
+ primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target
+ annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph
+ pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this
+ machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate
+ concept from `Target` objects.
+
+In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope
+is for this RFC to clear the way to taking on 5 in the future.
+
+Our proposal is:
+1. Extend `Target` to have a `DLDeviceType` attribute.
+2. Allow `Target` objects to be registered under a globally unique target label. Registration may be 'static' (ie
+ built into the TVM compiler via another REGISTER macro) and 'dynamic' (ie injected for a particular run of the
+ compiler, eg as part of `tvmc` command line processing). (This machinery should be reconciled with the existing
+ CUDA-specific target registration map.)
+3. Change the "on_device" call attributes to use a string instead of an integers (ie `DLDeviceType`). The string
+ can be of the form `<target label>` or `<target label>:<device id>`. The former simply implies a device id of 0.
+4. Rework device planning to use a pair of `Target` and 'device id' instead of `DLDeviceType`:
+ ```
+ class TargetDevice {
+ public:
+ Target target;
+ int device_id;
+ }
+ ```
+ (We could also use a `Device` and accept the redundant `DLDeviceType` specification.) It is trivial
+ to go from an "on_device" label to a `TargetDevice` and back using the global `Target` registry.
+5. Remove all uses of `TargetMap`. For example, in `LowerTEPass` we simply use the `TargetDevice` associated with
+ every primitive operator call already found by device planning.
+6. Bind two `TargetDevice`s as attributes on every `IRModule`:
+ - The default for primitive operators not otherwise constrained by "on_device" annotations.
Review comment:
I'll admit I've not thought at all about the leap from Target to Executor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml edited a comment on pull request #38: [RFC] Unified device/target/memory scope planning
Posted by GitBox <gi...@apache.org>.
mbs-octoml edited a comment on pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#issuecomment-948990025
Thanks for the comments @areusch and @manupa-arm . Now that I've started working on this (with an emphasis on handling memory scopes) I've decided to shift focus a bit. In particular Manupa I'm now much more motivated to tackle the BYOC/device planning overlap aspect, which I think you're particularly interested in. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml commented on pull request #38: [RFC] Improved multi-target handling
Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#issuecomment-932464732
Note to self: The With<Target> convention should probably also be removed by this work also, but I've not audited the code to see how pervasive it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml commented on pull request #38: [RFC] Unified device/target/memory scope planning
Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#issuecomment-948990025
Thanks for the comments @areusch . Now that I've started working on this (with an emphasis on handling memory scopes) I've decided to shift focus a bit. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml closed pull request #38: [RFC] Unified device/target/memory scope planning
Posted by GitBox <gi...@apache.org>.
mbs-octoml closed pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml edited a comment on pull request #38: [RFC] Improved multi-target handling
Posted by GitBox <gi...@apache.org>.
mbs-octoml edited a comment on pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#issuecomment-932464732
Note to self: The With<Target> convention should probably also be removed by this work also, but I've not audited the code to see how pervasive it is.
Target already has a `GetAttr<String>("device")` convention, but I think the map from name to `DLDeviceType` is only on the py side, and the attribute is not consistently set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml commented on a change in pull request #38: [RFC] Unified device/target/memory scope planning
Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on a change in pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#discussion_r734031103
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly
+ devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer
+ uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an
+ arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators.
+ Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system.
+3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for
+ primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the
+ overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to
+ deal with these different conventions.
+4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations
+ should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied
+ `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that
+ `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets,
+ it is particularly important to ensure there's a single source of truth for available `Target`s.
+5. TVM also supports a 'BYOC' extension mechanism. This allows `"target.<target name>"` annotations to be placed on
+ primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target
+ annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph
+ pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this
+ machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate
+ concept from `Target` objects.
+
+In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope
+is for this RFC to clear the way to taking on 5 in the future.
+
+Our proposal is:
+1. Extend `Target` to have a `DLDeviceType` attribute.
+2. Allow `Target` objects to be registered under a globally unique target label. Registration may be 'static' (ie
Review comment:
So for 'static' we already have the big-list-o-nvidia-targets (see target/tag.cc). I could imagine that's a pattern we keep building on. But as I started implementing this I quickly realized there's no need to introduce yet another name-to-thing binding mechanism for targets/devices etc.
- We already have the tag, so you can write Target("nividia/tesla-k40") and get that pre-defined target.
- If you want to programatically construct on_device annotations or anything else that needs to refer to specific targets, well, just use the host language to pass them around.
- If you want to write Relay script containing on_device annotations, use the metatable mechanism to bind to specific targets etc and refer to them by name in the text.
I've zapped all the target labels stuff from the RFC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml commented on a change in pull request #38: [RFC] Unified device/target/memory scope planning
Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on a change in pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#discussion_r734004042
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly
+ devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer
+ uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an
+ arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators.
+ Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system.
+3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for
+ primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the
+ overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to
+ deal with these different conventions.
+4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations
+ should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied
+ `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that
+ `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets,
+ it is particularly important to ensure there's a single source of truth for available `Target`s.
+5. TVM also supports a 'BYOC' extension mechanism. This allows `"target.<target name>"` annotations to be placed on
+ primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target
+ annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph
+ pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this
+ machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate
+ concept from `Target` objects.
+
+In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope
+is for this RFC to clear the way to taking on 5 in the future.
+
+Our proposal is:
+1. Extend `Target` to have a `DLDeviceType` attribute.
Review comment:
So I've rejigged to take on 5 and put less emphasis on the target wrangling which I think will work its self out by a combination of @Mousius work and incremental cleanup. Deep breath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml commented on a change in pull request #38: [RFC] Improved multi-target handling
Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on a change in pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#discussion_r725285510
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly
+ devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer
+ uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an
+ arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators.
+ Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system.
+3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for
+ primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the
+ overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to
+ deal with these different conventions.
+4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations
+ should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied
+ `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that
+ `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets,
+ it is particularly important to ensure there's a single source of truth for available `Target`s.
+5. TVM also supports a 'BYOC' extension mechanism. This allows `"target.<target name>"` annotations to be placed on
+ primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target
+ annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph
+ pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this
+ machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate
+ concept from `Target` objects.
+
+In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope
+is for this RFC to clear the way to taking on 5 in the future.
+
+Our proposal is:
+1. Extend `Target` to have a `DLDeviceType` attribute.
+2. Allow `Target` objects to be registered under a globally unique target label. Registration may be 'static' (ie
+ built into the TVM compiler via another REGISTER macro) and 'dynamic' (ie injected for a particular run of the
+ compiler, eg as part of `tvmc` command line processing). (This machinery should be reconciled with the existing
+ CUDA-specific target registration map.)
+3. Change the "on_device" call attributes to use a string instead of an integers (ie `DLDeviceType`). The string
+ can be of the form `<target label>` or `<target label>:<device id>`. The former simply implies a device id of 0.
+4. Rework device planning to use a pair of `Target` and 'device id' instead of `DLDeviceType`:
+ ```
+ class TargetDevice {
+ public:
+ Target target;
+ int device_id;
+ }
+ ```
+ (We could also use a `Device` and accept the redundant `DLDeviceType` specification.) It is trivial
+ to go from an "on_device" label to a `TargetDevice` and back using the global `Target` registry.
+5. Remove all uses of `TargetMap`. For example, in `LowerTEPass` we simply use the `TargetDevice` associated with
+ every primitive operator call already found by device planning.
+6. Bind two `TargetDevice`s as attributes on every `IRModule`:
+ - The default for primitive operators not otherwise constrained by "on_device" annotations.
+ - The default for non primitive operators, such as Relay control flow and shape computation.
+7. We remove the various copies of target/target_host reconciliation, `TargetMap`
+ construction and 'default/fallback' device calculation from the codebase.
+
+This proposal tackles the original problems:
+1. There's now no ambiguity about `Targets` since we propagate them from the global registry directly.
+2. We support device ids.
Review comment:
I'm going to call these `virtual_device_id`s, and leave how they are mapped to physical device ids either later in the compilation or at runtime unspecified. For example, the `virtual_device_id` could be nothing other than an index into an array of actual device structures at runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml commented on pull request #38: [RFC] Improved multi-target handling
Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#issuecomment-932274086
Thanks @manupa-arm for the reminder there were some good comments on #8892. I see a work stream:
1. get the multi-target handling under control, and stop relying on the limiting device type -> target mapping
2. allow device planning to be re-run with additional memory scope constraints to further partition, those constraints may originate in already lowered PrimFuncs.
3. allow labeling of sets of target/devices a la BYOC target labeling (but probably continue to just pick the 'first')
4. bring BYOC target labeling / partitioning and device planning together
5. replace naive partitioning with something subject to optimization
Here I want to just focus on 1 -- everything beyond that really needs face-to-face discussion.
From 2 onward obviously overlaps your USMP. My vague thought was we can work from opposite ends and reconcile at 5. Ie 2-4 sets us up to work in a combined Relay+TIR world, then 5 is where everything we've learned from USMP could perhaps be replayed. Anyway, that's just a vague thought so I'd love to talk more about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml commented on a change in pull request #38: [RFC] Unified device/target/memory scope planning
Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on a change in pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#discussion_r734032435
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly
+ devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer
+ uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an
+ arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators.
+ Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system.
+3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for
+ primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the
+ overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to
+ deal with these different conventions.
+4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations
+ should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied
+ `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that
+ `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets,
+ it is particularly important to ensure there's a single source of truth for available `Target`s.
+5. TVM also supports a 'BYOC' extension mechanism. This allows `"target.<target name>"` annotations to be placed on
+ primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target
+ annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph
+ pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this
+ machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate
+ concept from `Target` objects.
+
+In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope
+is for this RFC to clear the way to taking on 5 in the future.
+
+Our proposal is:
+1. Extend `Target` to have a `DLDeviceType` attribute.
+2. Allow `Target` objects to be registered under a globally unique target label. Registration may be 'static' (ie
+ built into the TVM compiler via another REGISTER macro) and 'dynamic' (ie injected for a particular run of the
+ compiler, eg as part of `tvmc` command line processing). (This machinery should be reconciled with the existing
+ CUDA-specific target registration map.)
+3. Change the "on_device" call attributes to use a string instead of an integers (ie `DLDeviceType`). The string
+ can be of the form `<target label>` or `<target label>:<device id>`. The former simply implies a device id of 0.
+4. Rework device planning to use a pair of `Target` and 'device id' instead of `DLDeviceType`:
+ ```
+ class TargetDevice {
+ public:
+ Target target;
+ int device_id;
+ }
+ ```
+ (We could also use a `Device` and accept the redundant `DLDeviceType` specification.) It is trivial
+ to go from an "on_device" label to a `TargetDevice` and back using the global `Target` registry.
+5. Remove all uses of `TargetMap`. For example, in `LowerTEPass` we simply use the `TargetDevice` associated with
Review comment:
I've removed this from the RFC. In #9313 you'll see I kept TargetMap but introduces a helper class to hid it. Over time I think we can replace TargetMap with just Array<Target>, but I feel it's not worth getting specific about that in an RFC and is more just a cleanup task. It may well come out of @Mousius ' work on tvmc target specification cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] areusch commented on a change in pull request #38: [RFC] Unified device/target/memory scope planning
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#discussion_r744917237
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
Review comment:
so is this constraining only the output of a particular subgraph (e.g. the subgraph can be actually implemented on a different device so long as a memory copy is done?)
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
+ should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc).
+2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay
+ sub-expression, including every primitive operator call. Sub-expressions which are unconstrained
+ are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data
+ needs to cross device boundaries.
+3. The user must also supply a list of `Target` objects. The compiler uses that list to build
+ a `TargetMap` from `DLDeviceType` to `Target`.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile
+ ('lower') that primitive for that device. The `Target` to use for that compilation is found from
+ the `TargetMap` by the `LowerTEPass`.
+
+For the BYOC flow things are quite different:
+1. Operators may be annotated with an `FTVMAnnotateTarget` function for a particular
+ `target.<name>`. Here `<name>` serves only to distinguish possible BYOC toolchain names and is
+ currently not connected to the `Target` machinery in any way. The function should return true if
+ the given expression could be compiled for toolchain `<name>`. (However there are currently no
+ examples of this annotation in-tree.)
+2. The `MergeComposite` pass can be used to assign a `"Composite"` attribute to Relay functions
+ which have been hoisted out of a larger expression based on a fusion pattern. The attribute can
+ have any value of the form `"some.arbitrary.prefix.<name>"`. Again, this indicates the function
+ could be compiled for toolchain `<name>`. (The EthosU compilation flow illustrates this approach
+ in-tree.)
+3. The `AnnotateTarget` pass looks for the annotations from (1) and (2) to decide the unique
+ toolchain name for every Relay sub-expression which should go via a BYOC path. The transitions in
+ to and out of those sub-expressions are marked with `compiler_begin` and `compiler_end`
Review comment:
just curious, because i've seen compiler_begin and compiler_end before but not many examples in complex programs: are these essentially a source-level annotation e.g. marking all Relay expressions between the two annotations as offloaded to a particular compiler? why shouldn't these be hierarchical e.g. CompilerBlock which contains the subgraph as a tree?
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
+ should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc).
+2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay
+ sub-expression, including every primitive operator call. Sub-expressions which are unconstrained
+ are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data
+ needs to cross device boundaries.
+3. The user must also supply a list of `Target` objects. The compiler uses that list to build
+ a `TargetMap` from `DLDeviceType` to `Target`.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile
+ ('lower') that primitive for that device. The `Target` to use for that compilation is found from
+ the `TargetMap` by the `LowerTEPass`.
+
+For the BYOC flow things are quite different:
+1. Operators may be annotated with an `FTVMAnnotateTarget` function for a particular
+ `target.<name>`. Here `<name>` serves only to distinguish possible BYOC toolchain names and is
+ currently not connected to the `Target` machinery in any way. The function should return true if
+ the given expression could be compiled for toolchain `<name>`. (However there are currently no
+ examples of this annotation in-tree.)
+2. The `MergeComposite` pass can be used to assign a `"Composite"` attribute to Relay functions
+ which have been hoisted out of a larger expression based on a fusion pattern. The attribute can
+ have any value of the form `"some.arbitrary.prefix.<name>"`. Again, this indicates the function
+ could be compiled for toolchain `<name>`. (The EthosU compilation flow illustrates this approach
+ in-tree.)
+3. The `AnnotateTarget` pass looks for the annotations from (1) and (2) to decide the unique
+ toolchain name for every Relay sub-expression which should go via a BYOC path. The transitions in
+ to and out of those sub-expressions are marked with `compiler_begin` and `compiler_end`
+ annotations.
+4. The `PartitionGraph` pass hoists sub-expressions delimited by `compiler_begin` and `compiler_end`
+ annotations into new top-level `Function`s with a `"Compiler"` attribute bound to the toolchain
+ `<name>`.
+5. The rest of the compilation flow treats `"Compiler"` annotated functions specially.
+
+We have 6 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple
+ tensor-friendly devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a
+ `DLDeviceType` no longer uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a
+ `DLDeviceType` and an arbitrary 'device id', TVM does not consistently plumb the device id
+ through annotations, passes and operators. Thus currently we cannot use 'device id' to
+ distinguish, eg, two CPUs in the same system.
+3. Upcoming work requires us to distinguish and propagate memory scopes for data at the Relay
+ level. (See also [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md)
+ which has a similar need for memory scope propagation at the TIR level). This is an identical
+ problem to propagating devices, and it seems most natural to simply combine targets, devices and
+ memory scopes into a single 'target of device planing' rather than implementing a whole new pass.
+4. Device planning currently has no machinery to hoist adjacent expressions which share the same device
+ into their own Relay `Function`. For all our executors except VM that's unnecessary anyway since
+ all Relay expressions left over after lowering are interpreted by the runtime. However for AOT we
+ have to compile *all* Relay code for a particular target. Note the BOYC machinery does support this,
+ but for the purposes of redirecting the compilation flow entirely. We need a middle ground.
+5. The BYOC flow is not connected to the `Target` machinery in any way.
+6. The BYOC annotate/partition flow is very similar to the device annotate/rewrite flow. For comparison:
+
+ | Feature | Device Planning | BYOC |
+ | --------------------- | -------------------------- | ----------------------------------------------- |
+ | Source of annotations | `on_device`, `device_copy` | `FTVMAnnotateTarget`, `MergeComposite`+patterns |
+ | Target of planning | DLDeviceType | Toolchain name |
+ | Propagation | Unification based | Ad-hoc |
+ | Relay support | Full | First-order, no ADTs |
+ | Delimiting | insert `device_copy` | insert `compiler_begin`, `compiler_end` |
+ | Multiple per expr | No | Yes (though always picks first) |
+ | Hoists into functions | No | Yes |
+ | Customized heuristics | No | No |
+
+ Taking the 'upper bound' of the two implementations seems ideal, especially to address issues 4 (limitation
+ of device planning) and 5 (limitation of BYOC) above.
+
+Our proposal is:
+1. We introduce a new FFI-friendly class to represent a *S*torage or *E*xecution *Scope*:
+
+ ```
+ class SEScope {
+ DLDeviceType device_type;
+ int virtual_device_id;
+ Target target;
+ String memory_scope;
+ }
+ ```
+
+ We allow each of these fields to be independently 'constrained' (ie have a specific value) or
+ 'unconstrained' (no specific value for the field is known yet). In particular, it is valid for
+ an `SEScope` to contain only a `device_type`. However if the `target` field is defined then
+ `device_type` must equal `target->kind->device_type`.
+
+2. At this stage we leave the `memory_scope` field uninterpreted. For example, we don't attempt to
+ represent that, eg, `"global"` on a `kDLCPU` is the same memory area as `"host"` on a `kDLCUDA` and thus no
+ `device_copy` operation is required between those scopes. We'll pick this issue up again after
+ [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md)
+ has landed.
+
+3. The `on_device` and `device_copy` call attributes use `SEScope`s instead of integers. However the Python
+ bindings for these 'operators' continue to accept a `Device` for convenience. The machinery in `LowerTEPass`
+ which resolves `DLDeviceTypes` to `Targets` is moved up in the compilation flow and becomes part of
+ `PlanDevices`. In particular, any `SEScope` encountered during device planning is 'canonicalized' to fill
+ in a `Target` by the same lookup as we do today. This means we continue to support the easy shorthand of
+ referring to devices by the `DLDeviceType` alone. However, advanced users can supply a `SEScope` to these
+ operators which contains the exact `Target` to use.
+
+4. We rework device planning to be in terms of `SEScope`s instead of `DLDeviceTypes`. Two `SEScope`s
+ become special:
+ - We need a default scope for all primitive operators which are not otherwise
+ constrained to a particular scope.
+ - We need a scope for 'host-only' operations and data, such as for shapes and shape functions.
+ (Currently this is hardcoded to `kDLCPU`).
+
+5. We extend `PlanDevices` to be able to a) run *after* lowering and b) refine existing constraints. It will
+ look inside calls to `PrimFunc`s and follow the chain:
+
+ ```
+ tir::PrimFunc.buffer_map -> tir::Buffer.data -> tir::Var.type_annotation -> PointerType.storage_scope -> String
+ ```
+
+ to discover the memory scope for each Relay argument. That scope will enter `SEScope`s and flow through the
+ existing unification machinery. The existing sub-pass in `PlanDevices` will insert `device_copy` calls
+ wherever sub-expressions disagree on their memory scope.
+
+ (An additional pass is planned to heuristically move `device_copy`s around, and eliminate redundant
+ copies, however that's outside the scope of this RFC.)
+
+6. We rework `PartitionGraph` to `PartitionBySEScope` to work on `SEScope` annotations instead of
+ `compiler_begin` and `compiler_end` annotations. Algorithmically it's not a big change -- maximal
+ sub-expressions which share the same `SEScope` (or a projection thereof, eg just the `target`) are hoisted
+ into global `Function`s. The function's `"result_se_scope"` attribute describes both the scope holding the
+ function's result *and* the `Target` for which the function is to be compiled.
+
+7. We allow `MergeComposite` to be used to insert `on_device` annotations, call it `MergeAndAnnotate`.
+
+8. (?) We rework `AnnotateTarget` to just look for `FTVMAnnotateTarget` operator attributes, call it
+ `AnnotateSEScopes`. When the function fires an `on_device` annotation is inserted. However since
Review comment:
clarifying my understanding:
```suggestion
`AnnotateSEScopes`. When `FTVMAnnotateSEScopes` returns true, an `on_device` annotation is inserted. However since
```
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
+ should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc).
+2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay
+ sub-expression, including every primitive operator call. Sub-expressions which are unconstrained
+ are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data
+ needs to cross device boundaries.
+3. The user must also supply a list of `Target` objects. The compiler uses that list to build
+ a `TargetMap` from `DLDeviceType` to `Target`.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile
+ ('lower') that primitive for that device. The `Target` to use for that compilation is found from
+ the `TargetMap` by the `LowerTEPass`.
+
+For the BYOC flow things are quite different:
+1. Operators may be annotated with an `FTVMAnnotateTarget` function for a particular
+ `target.<name>`. Here `<name>` serves only to distinguish possible BYOC toolchain names and is
+ currently not connected to the `Target` machinery in any way. The function should return true if
+ the given expression could be compiled for toolchain `<name>`. (However there are currently no
+ examples of this annotation in-tree.)
+2. The `MergeComposite` pass can be used to assign a `"Composite"` attribute to Relay functions
+ which have been hoisted out of a larger expression based on a fusion pattern. The attribute can
+ have any value of the form `"some.arbitrary.prefix.<name>"`. Again, this indicates the function
+ could be compiled for toolchain `<name>`. (The EthosU compilation flow illustrates this approach
+ in-tree.)
+3. The `AnnotateTarget` pass looks for the annotations from (1) and (2) to decide the unique
+ toolchain name for every Relay sub-expression which should go via a BYOC path. The transitions in
+ to and out of those sub-expressions are marked with `compiler_begin` and `compiler_end`
+ annotations.
+4. The `PartitionGraph` pass hoists sub-expressions delimited by `compiler_begin` and `compiler_end`
+ annotations into new top-level `Function`s with a `"Compiler"` attribute bound to the toolchain
+ `<name>`.
+5. The rest of the compilation flow treats `"Compiler"` annotated functions specially.
+
+We have 6 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple
+ tensor-friendly devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a
+ `DLDeviceType` no longer uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a
+ `DLDeviceType` and an arbitrary 'device id', TVM does not consistently plumb the device id
+ through annotations, passes and operators. Thus currently we cannot use 'device id' to
+ distinguish, eg, two CPUs in the same system.
+3. Upcoming work requires us to distinguish and propagate memory scopes for data at the Relay
+ level. (See also [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md)
+ which has a similar need for memory scope propagation at the TIR level). This is an identical
+ problem to propagating devices, and it seems most natural to simply combine targets, devices and
+ memory scopes into a single 'target of device planing' rather than implementing a whole new pass.
+4. Device planning currently has no machinery to hoist adjacent expressions which share the same device
+ into their own Relay `Function`. For all our executors except VM that's unnecessary anyway since
+ all Relay expressions left over after lowering are interpreted by the runtime. However for AOT we
+ have to compile *all* Relay code for a particular target. Note the BOYC machinery does support this,
+ but for the purposes of redirecting the compilation flow entirely. We need a middle ground.
+5. The BYOC flow is not connected to the `Target` machinery in any way.
+6. The BYOC annotate/partition flow is very similar to the device annotate/rewrite flow. For comparison:
+
+ | Feature | Device Planning | BYOC |
+ | --------------------- | -------------------------- | ----------------------------------------------- |
+ | Source of annotations | `on_device`, `device_copy` | `FTVMAnnotateTarget`, `MergeComposite`+patterns |
+ | Target of planning | DLDeviceType | Toolchain name |
+ | Propagation | Unification based | Ad-hoc |
+ | Relay support | Full | First-order, no ADTs |
+ | Delimiting | insert `device_copy` | insert `compiler_begin`, `compiler_end` |
+ | Multiple per expr | No | Yes (though always picks first) |
+ | Hoists into functions | No | Yes |
+ | Customized heuristics | No | No |
+
+ Taking the 'upper bound' of the two implementations seems ideal, especially to address issues 4 (limitation
+ of device planning) and 5 (limitation of BYOC) above.
+
+Our proposal is:
+1. We introduce a new FFI-friendly class to represent a *S*torage or *E*xecution *Scope*:
+
+ ```
+ class SEScope {
+ DLDeviceType device_type;
+ int virtual_device_id;
+ Target target;
+ String memory_scope;
+ }
+ ```
+
+ We allow each of these fields to be independently 'constrained' (ie have a specific value) or
+ 'unconstrained' (no specific value for the field is known yet). In particular, it is valid for
+ an `SEScope` to contain only a `device_type`. However if the `target` field is defined then
+ `device_type` must equal `target->kind->device_type`.
+
+2. At this stage we leave the `memory_scope` field uninterpreted. For example, we don't attempt to
+ represent that, eg, `"global"` on a `kDLCPU` is the same memory area as `"host"` on a `kDLCUDA` and thus no
+ `device_copy` operation is required between those scopes. We'll pick this issue up again after
+ [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md)
+ has landed.
+
+3. The `on_device` and `device_copy` call attributes use `SEScope`s instead of integers. However the Python
+ bindings for these 'operators' continue to accept a `Device` for convenience. The machinery in `LowerTEPass`
+ which resolves `DLDeviceTypes` to `Targets` is moved up in the compilation flow and becomes part of
+ `PlanDevices`. In particular, any `SEScope` encountered during device planning is 'canonicalized' to fill
+ in a `Target` by the same lookup as we do today. This means we continue to support the easy shorthand of
+ referring to devices by the `DLDeviceType` alone. However, advanced users can supply a `SEScope` to these
+ operators which contains the exact `Target` to use.
Review comment:
what would be roughly the deprecation plan here? eventually we ban all the inputs to the compiler which could refer to SEScope in terms of DLDeviceType and then tighten the typing requirements here? this would be a backwards-incompatible Relay change. cc @jroesch
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
+ should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc).
+2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay
+ sub-expression, including every primitive operator call. Sub-expressions which are unconstrained
+ are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data
+ needs to cross device boundaries.
+3. The user must also supply a list of `Target` objects. The compiler uses that list to build
+ a `TargetMap` from `DLDeviceType` to `Target`.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile
+ ('lower') that primitive for that device. The `Target` to use for that compilation is found from
+ the `TargetMap` by the `LowerTEPass`.
+
+For the BYOC flow things are quite different:
+1. Operators may be annotated with an `FTVMAnnotateTarget` function for a particular
+ `target.<name>`. Here `<name>` serves only to distinguish possible BYOC toolchain names and is
+ currently not connected to the `Target` machinery in any way. The function should return true if
+ the given expression could be compiled for toolchain `<name>`. (However there are currently no
+ examples of this annotation in-tree.)
+2. The `MergeComposite` pass can be used to assign a `"Composite"` attribute to Relay functions
+ which have been hoisted out of a larger expression based on a fusion pattern. The attribute can
+ have any value of the form `"some.arbitrary.prefix.<name>"`. Again, this indicates the function
+ could be compiled for toolchain `<name>`. (The EthosU compilation flow illustrates this approach
+ in-tree.)
+3. The `AnnotateTarget` pass looks for the annotations from (1) and (2) to decide the unique
+ toolchain name for every Relay sub-expression which should go via a BYOC path. The transitions in
+ to and out of those sub-expressions are marked with `compiler_begin` and `compiler_end`
+ annotations.
+4. The `PartitionGraph` pass hoists sub-expressions delimited by `compiler_begin` and `compiler_end`
+ annotations into new top-level `Function`s with a `"Compiler"` attribute bound to the toolchain
+ `<name>`.
+5. The rest of the compilation flow treats `"Compiler"` annotated functions specially.
+
+We have 6 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple
+ tensor-friendly devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a
+ `DLDeviceType` no longer uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a
+ `DLDeviceType` and an arbitrary 'device id', TVM does not consistently plumb the device id
+ through annotations, passes and operators. Thus currently we cannot use 'device id' to
+ distinguish, eg, two CPUs in the same system.
+3. Upcoming work requires us to distinguish and propagate memory scopes for data at the Relay
+ level. (See also [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md)
+ which has a similar need for memory scope propagation at the TIR level). This is an identical
+ problem to propagating devices, and it seems most natural to simply combine targets, devices and
+ memory scopes into a single 'target of device planing' rather than implementing a whole new pass.
+4. Device planning currently has no machinery to hoist adjacent expressions which share the same device
+ into their own Relay `Function`. For all our executors except VM that's unnecessary anyway since
+ all Relay expressions left over after lowering are interpreted by the runtime. However for AOT we
+ have to compile *all* Relay code for a particular target. Note the BOYC machinery does support this,
+ but for the purposes of redirecting the compilation flow entirely. We need a middle ground.
Review comment:
```suggestion
have to compile *all* Relay code for a particular target. Note the BYOC machinery does support this,
but for the purposes of more accurately modeling offloaded compute in the main compilation flow, we need a middle ground.
```
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
+ should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc).
+2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay
+ sub-expression, including every primitive operator call. Sub-expressions which are unconstrained
+ are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data
+ needs to cross device boundaries.
+3. The user must also supply a list of `Target` objects. The compiler uses that list to build
Review comment:
would be good to clarify as they are also required at runtime to the executor ctor
```suggestion
3. The user must also supply a list of `Target` objects to `tvm.relay.build`. The compiler uses that list to build
```
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
Review comment:
nit: sequentially is a bit misleading--maybe suggest
```suggestion
TVM supports 'hetrogeneous' execution, whereby primitive operators may be evaluated (in topological order)
```
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
+ should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc).
+2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay
+ sub-expression, including every primitive operator call. Sub-expressions which are unconstrained
+ are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data
Review comment:
"default" also is called "fallback," right?
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
+ should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc).
+2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay
+ sub-expression, including every primitive operator call. Sub-expressions which are unconstrained
+ are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data
+ needs to cross device boundaries.
+3. The user must also supply a list of `Target` objects. The compiler uses that list to build
+ a `TargetMap` from `DLDeviceType` to `Target`.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile
+ ('lower') that primitive for that device. The `Target` to use for that compilation is found from
+ the `TargetMap` by the `LowerTEPass`.
+
+For the BYOC flow things are quite different:
+1. Operators may be annotated with an `FTVMAnnotateTarget` function for a particular
+ `target.<name>`. Here `<name>` serves only to distinguish possible BYOC toolchain names and is
+ currently not connected to the `Target` machinery in any way. The function should return true if
+ the given expression could be compiled for toolchain `<name>`. (However there are currently no
+ examples of this annotation in-tree.)
+2. The `MergeComposite` pass can be used to assign a `"Composite"` attribute to Relay functions
+ which have been hoisted out of a larger expression based on a fusion pattern. The attribute can
+ have any value of the form `"some.arbitrary.prefix.<name>"`. Again, this indicates the function
+ could be compiled for toolchain `<name>`. (The EthosU compilation flow illustrates this approach
+ in-tree.)
+3. The `AnnotateTarget` pass looks for the annotations from (1) and (2) to decide the unique
+ toolchain name for every Relay sub-expression which should go via a BYOC path. The transitions in
+ to and out of those sub-expressions are marked with `compiler_begin` and `compiler_end`
+ annotations.
+4. The `PartitionGraph` pass hoists sub-expressions delimited by `compiler_begin` and `compiler_end`
+ annotations into new top-level `Function`s with a `"Compiler"` attribute bound to the toolchain
+ `<name>`.
+5. The rest of the compilation flow treats `"Compiler"` annotated functions specially.
+
+We have 6 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple
+ tensor-friendly devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a
+ `DLDeviceType` no longer uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a
+ `DLDeviceType` and an arbitrary 'device id', TVM does not consistently plumb the device id
+ through annotations, passes and operators. Thus currently we cannot use 'device id' to
+ distinguish, eg, two CPUs in the same system.
+3. Upcoming work requires us to distinguish and propagate memory scopes for data at the Relay
+ level. (See also [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md)
+ which has a similar need for memory scope propagation at the TIR level). This is an identical
+ problem to propagating devices, and it seems most natural to simply combine targets, devices and
+ memory scopes into a single 'target of device planing' rather than implementing a whole new pass.
Review comment:
i agree, but just clarifying we aren't using a single identifier to describe both the device and the memory scope?
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
+ should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc).
+2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay
+ sub-expression, including every primitive operator call. Sub-expressions which are unconstrained
+ are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data
+ needs to cross device boundaries.
+3. The user must also supply a list of `Target` objects. The compiler uses that list to build
+ a `TargetMap` from `DLDeviceType` to `Target`.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile
+ ('lower') that primitive for that device. The `Target` to use for that compilation is found from
+ the `TargetMap` by the `LowerTEPass`.
+
+For the BYOC flow things are quite different:
+1. Operators may be annotated with an `FTVMAnnotateTarget` function for a particular
+ `target.<name>`. Here `<name>` serves only to distinguish possible BYOC toolchain names and is
+ currently not connected to the `Target` machinery in any way. The function should return true if
+ the given expression could be compiled for toolchain `<name>`. (However there are currently no
+ examples of this annotation in-tree.)
+2. The `MergeComposite` pass can be used to assign a `"Composite"` attribute to Relay functions
+ which have been hoisted out of a larger expression based on a fusion pattern. The attribute can
+ have any value of the form `"some.arbitrary.prefix.<name>"`. Again, this indicates the function
+ could be compiled for toolchain `<name>`. (The EthosU compilation flow illustrates this approach
+ in-tree.)
+3. The `AnnotateTarget` pass looks for the annotations from (1) and (2) to decide the unique
+ toolchain name for every Relay sub-expression which should go via a BYOC path. The transitions in
+ to and out of those sub-expressions are marked with `compiler_begin` and `compiler_end`
+ annotations.
+4. The `PartitionGraph` pass hoists sub-expressions delimited by `compiler_begin` and `compiler_end`
+ annotations into new top-level `Function`s with a `"Compiler"` attribute bound to the toolchain
+ `<name>`.
+5. The rest of the compilation flow treats `"Compiler"` annotated functions specially.
+
+We have 6 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple
+ tensor-friendly devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a
+ `DLDeviceType` no longer uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a
+ `DLDeviceType` and an arbitrary 'device id', TVM does not consistently plumb the device id
+ through annotations, passes and operators. Thus currently we cannot use 'device id' to
+ distinguish, eg, two CPUs in the same system.
+3. Upcoming work requires us to distinguish and propagate memory scopes for data at the Relay
+ level. (See also [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md)
+ which has a similar need for memory scope propagation at the TIR level). This is an identical
+ problem to propagating devices, and it seems most natural to simply combine targets, devices and
+ memory scopes into a single 'target of device planing' rather than implementing a whole new pass.
+4. Device planning currently has no machinery to hoist adjacent expressions which share the same device
+ into their own Relay `Function`. For all our executors except VM that's unnecessary anyway since
+ all Relay expressions left over after lowering are interpreted by the runtime. However for AOT we
+ have to compile *all* Relay code for a particular target. Note the BOYC machinery does support this,
+ but for the purposes of redirecting the compilation flow entirely. We need a middle ground.
+5. The BYOC flow is not connected to the `Target` machinery in any way.
+6. The BYOC annotate/partition flow is very similar to the device annotate/rewrite flow. For comparison:
+
+ | Feature | Device Planning | BYOC |
+ | --------------------- | -------------------------- | ----------------------------------------------- |
+ | Source of annotations | `on_device`, `device_copy` | `FTVMAnnotateTarget`, `MergeComposite`+patterns |
+ | Target of planning | DLDeviceType | Toolchain name |
+ | Propagation | Unification based | Ad-hoc |
+ | Relay support | Full | First-order, no ADTs |
+ | Delimiting | insert `device_copy` | insert `compiler_begin`, `compiler_end` |
+ | Multiple per expr | No | Yes (though always picks first) |
+ | Hoists into functions | No | Yes |
+ | Customized heuristics | No | No |
+
+ Taking the 'upper bound' of the two implementations seems ideal, especially to address issues 4 (limitation
+ of device planning) and 5 (limitation of BYOC) above.
+
+Our proposal is:
+1. We introduce a new FFI-friendly class to represent a *S*torage or *E*xecution *Scope*:
+
+ ```
+ class SEScope {
+ DLDeviceType device_type;
+ int virtual_device_id;
+ Target target;
+ String memory_scope;
+ }
+ ```
+
+ We allow each of these fields to be independently 'constrained' (ie have a specific value) or
+ 'unconstrained' (no specific value for the field is known yet). In particular, it is valid for
+ an `SEScope` to contain only a `device_type`. However if the `target` field is defined then
+ `device_type` must equal `target->kind->device_type`.
+
+2. At this stage we leave the `memory_scope` field uninterpreted. For example, we don't attempt to
+ represent that, eg, `"global"` on a `kDLCPU` is the same memory area as `"host"` on a `kDLCUDA` and thus no
+ `device_copy` operation is required between those scopes. We'll pick this issue up again after
+ [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md)
+ has landed.
+
+3. The `on_device` and `device_copy` call attributes use `SEScope`s instead of integers. However the Python
+ bindings for these 'operators' continue to accept a `Device` for convenience. The machinery in `LowerTEPass`
+ which resolves `DLDeviceTypes` to `Targets` is moved up in the compilation flow and becomes part of
+ `PlanDevices`. In particular, any `SEScope` encountered during device planning is 'canonicalized' to fill
+ in a `Target` by the same lookup as we do today. This means we continue to support the easy shorthand of
+ referring to devices by the `DLDeviceType` alone. However, advanced users can supply a `SEScope` to these
+ operators which contains the exact `Target` to use.
+
+4. We rework device planning to be in terms of `SEScope`s instead of `DLDeviceTypes`. Two `SEScope`s
+ become special:
+ - We need a default scope for all primitive operators which are not otherwise
+ constrained to a particular scope.
+ - We need a scope for 'host-only' operations and data, such as for shapes and shape functions.
+ (Currently this is hardcoded to `kDLCPU`).
+
+5. We extend `PlanDevices` to be able to a) run *after* lowering and b) refine existing constraints. It will
+ look inside calls to `PrimFunc`s and follow the chain:
+
+ ```
+ tir::PrimFunc.buffer_map -> tir::Buffer.data -> tir::Var.type_annotation -> PointerType.storage_scope -> String
+ ```
+
+ to discover the memory scope for each Relay argument. That scope will enter `SEScope`s and flow through the
Review comment:
what do you mean by "enter `SEScope`s"?
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
+ should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc).
+2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay
+ sub-expression, including every primitive operator call. Sub-expressions which are unconstrained
+ are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data
+ needs to cross device boundaries.
+3. The user must also supply a list of `Target` objects. The compiler uses that list to build
+ a `TargetMap` from `DLDeviceType` to `Target`.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile
+ ('lower') that primitive for that device. The `Target` to use for that compilation is found from
+ the `TargetMap` by the `LowerTEPass`.
+
+For the BYOC flow things are quite different:
+1. Operators may be annotated with an `FTVMAnnotateTarget` function for a particular
+ `target.<name>`. Here `<name>` serves only to distinguish possible BYOC toolchain names and is
+ currently not connected to the `Target` machinery in any way. The function should return true if
+ the given expression could be compiled for toolchain `<name>`. (However there are currently no
+ examples of this annotation in-tree.)
+2. The `MergeComposite` pass can be used to assign a `"Composite"` attribute to Relay functions
+ which have been hoisted out of a larger expression based on a fusion pattern. The attribute can
+ have any value of the form `"some.arbitrary.prefix.<name>"`. Again, this indicates the function
+ could be compiled for toolchain `<name>`. (The EthosU compilation flow illustrates this approach
+ in-tree.)
+3. The `AnnotateTarget` pass looks for the annotations from (1) and (2) to decide the unique
+ toolchain name for every Relay sub-expression which should go via a BYOC path. The transitions in
+ to and out of those sub-expressions are marked with `compiler_begin` and `compiler_end`
+ annotations.
+4. The `PartitionGraph` pass hoists sub-expressions delimited by `compiler_begin` and `compiler_end`
+ annotations into new top-level `Function`s with a `"Compiler"` attribute bound to the toolchain
+ `<name>`.
+5. The rest of the compilation flow treats `"Compiler"` annotated functions specially.
+
+We have 6 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple
+ tensor-friendly devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a
+ `DLDeviceType` no longer uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a
+ `DLDeviceType` and an arbitrary 'device id', TVM does not consistently plumb the device id
+ through annotations, passes and operators. Thus currently we cannot use 'device id' to
+ distinguish, eg, two CPUs in the same system.
+3. Upcoming work requires us to distinguish and propagate memory scopes for data at the Relay
+ level. (See also [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md)
+ which has a similar need for memory scope propagation at the TIR level). This is an identical
+ problem to propagating devices, and it seems most natural to simply combine targets, devices and
+ memory scopes into a single 'target of device planing' rather than implementing a whole new pass.
+4. Device planning currently has no machinery to hoist adjacent expressions which share the same device
+ into their own Relay `Function`. For all our executors except VM that's unnecessary anyway since
+ all Relay expressions left over after lowering are interpreted by the runtime. However for AOT we
+ have to compile *all* Relay code for a particular target. Note the BOYC machinery does support this,
+ but for the purposes of redirecting the compilation flow entirely. We need a middle ground.
+5. The BYOC flow is not connected to the `Target` machinery in any way.
+6. The BYOC annotate/partition flow is very similar to the device annotate/rewrite flow. For comparison:
+
+ | Feature | Device Planning | BYOC |
+ | --------------------- | -------------------------- | ----------------------------------------------- |
+ | Source of annotations | `on_device`, `device_copy` | `FTVMAnnotateTarget`, `MergeComposite`+patterns |
+ | Target of planning | DLDeviceType | Toolchain name |
+ | Propagation | Unification based | Ad-hoc |
+ | Relay support | Full | First-order, no ADTs |
+ | Delimiting | insert `device_copy` | insert `compiler_begin`, `compiler_end` |
+ | Multiple per expr | No | Yes (though always picks first) |
+ | Hoists into functions | No | Yes |
+ | Customized heuristics | No | No |
+
+ Taking the 'upper bound' of the two implementations seems ideal, especially to address issues 4 (limitation
+ of device planning) and 5 (limitation of BYOC) above.
+
+Our proposal is:
+1. We introduce a new FFI-friendly class to represent a *S*torage or *E*xecution *Scope*:
+
+ ```
+ class SEScope {
+ DLDeviceType device_type;
+ int virtual_device_id;
+ Target target;
+ String memory_scope;
+ }
+ ```
+
+ We allow each of these fields to be independently 'constrained' (ie have a specific value) or
+ 'unconstrained' (no specific value for the field is known yet). In particular, it is valid for
+ an `SEScope` to contain only a `device_type`. However if the `target` field is defined then
+ `device_type` must equal `target->kind->device_type`.
+
+2. At this stage we leave the `memory_scope` field uninterpreted. For example, we don't attempt to
+ represent that, eg, `"global"` on a `kDLCPU` is the same memory area as `"host"` on a `kDLCUDA` and thus no
+ `device_copy` operation is required between those scopes. We'll pick this issue up again after
+ [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md)
+ has landed.
+
+3. The `on_device` and `device_copy` call attributes use `SEScope`s instead of integers. However the Python
+ bindings for these 'operators' continue to accept a `Device` for convenience. The machinery in `LowerTEPass`
+ which resolves `DLDeviceTypes` to `Targets` is moved up in the compilation flow and becomes part of
+ `PlanDevices`. In particular, any `SEScope` encountered during device planning is 'canonicalized' to fill
+ in a `Target` by the same lookup as we do today. This means we continue to support the easy shorthand of
+ referring to devices by the `DLDeviceType` alone. However, advanced users can supply a `SEScope` to these
+ operators which contains the exact `Target` to use.
+
+4. We rework device planning to be in terms of `SEScope`s instead of `DLDeviceTypes`. Two `SEScope`s
+ become special:
+ - We need a default scope for all primitive operators which are not otherwise
+ constrained to a particular scope.
+ - We need a scope for 'host-only' operations and data, such as for shapes and shape functions.
+ (Currently this is hardcoded to `kDLCPU`).
+
+5. We extend `PlanDevices` to be able to a) run *after* lowering and b) refine existing constraints. It will
+ look inside calls to `PrimFunc`s and follow the chain:
+
+ ```
+ tir::PrimFunc.buffer_map -> tir::Buffer.data -> tir::Var.type_annotation -> PointerType.storage_scope -> String
+ ```
+
+ to discover the memory scope for each Relay argument. That scope will enter `SEScope`s and flow through the
+ existing unification machinery. The existing sub-pass in `PlanDevices` will insert `device_copy` calls
+ wherever sub-expressions disagree on their memory scope.
+
+ (An additional pass is planned to heuristically move `device_copy`s around, and eliminate redundant
+ copies, however that's outside the scope of this RFC.)
+
+6. We rework `PartitionGraph` to `PartitionBySEScope` to work on `SEScope` annotations instead of
+ `compiler_begin` and `compiler_end` annotations. Algorithmically it's not a big change -- maximal
+ sub-expressions which share the same `SEScope` (or a projection thereof, eg just the `target`) are hoisted
+ into global `Function`s. The function's `"result_se_scope"` attribute describes both the scope holding the
Review comment:
so then here, this sort of implements the "grouping adjacent expressions onto the same device" as a side-effect?
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
+ should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc).
+2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay
+ sub-expression, including every primitive operator call. Sub-expressions which are unconstrained
+ are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data
+ needs to cross device boundaries.
+3. The user must also supply a list of `Target` objects. The compiler uses that list to build
+ a `TargetMap` from `DLDeviceType` to `Target`.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile
+ ('lower') that primitive for that device. The `Target` to use for that compilation is found from
+ the `TargetMap` by the `LowerTEPass`.
+
+For the BYOC flow things are quite different:
+1. Operators may be annotated with an `FTVMAnnotateTarget` function for a particular
+ `target.<name>`. Here `<name>` serves only to distinguish possible BYOC toolchain names and is
+ currently not connected to the `Target` machinery in any way. The function should return true if
+ the given expression could be compiled for toolchain `<name>`. (However there are currently no
+ examples of this annotation in-tree.)
+2. The `MergeComposite` pass can be used to assign a `"Composite"` attribute to Relay functions
+ which have been hoisted out of a larger expression based on a fusion pattern. The attribute can
+ have any value of the form `"some.arbitrary.prefix.<name>"`. Again, this indicates the function
+ could be compiled for toolchain `<name>`. (The EthosU compilation flow illustrates this approach
+ in-tree.)
+3. The `AnnotateTarget` pass looks for the annotations from (1) and (2) to decide the unique
+ toolchain name for every Relay sub-expression which should go via a BYOC path. The transitions in
+ to and out of those sub-expressions are marked with `compiler_begin` and `compiler_end`
+ annotations.
+4. The `PartitionGraph` pass hoists sub-expressions delimited by `compiler_begin` and `compiler_end`
+ annotations into new top-level `Function`s with a `"Compiler"` attribute bound to the toolchain
+ `<name>`.
+5. The rest of the compilation flow treats `"Compiler"` annotated functions specially.
+
+We have 6 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple
+ tensor-friendly devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a
+ `DLDeviceType` no longer uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a
+ `DLDeviceType` and an arbitrary 'device id', TVM does not consistently plumb the device id
+ through annotations, passes and operators. Thus currently we cannot use 'device id' to
+ distinguish, eg, two CPUs in the same system.
+3. Upcoming work requires us to distinguish and propagate memory scopes for data at the Relay
+ level. (See also [RFC #9](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0009_Unified_Static_Memory_Planning.md)
+ which has a similar need for memory scope propagation at the TIR level). This is an identical
+ problem to propagating devices, and it seems most natural to simply combine targets, devices and
+ memory scopes into a single 'target of device planing' rather than implementing a whole new pass.
+4. Device planning currently has no machinery to hoist adjacent expressions which share the same device
+ into their own Relay `Function`. For all our executors except VM that's unnecessary anyway since
+ all Relay expressions left over after lowering are interpreted by the runtime. However for AOT we
+ have to compile *all* Relay code for a particular target. Note the BOYC machinery does support this,
+ but for the purposes of redirecting the compilation flow entirely. We need a middle ground.
+5. The BYOC flow is not connected to the `Target` machinery in any way.
+6. The BYOC annotate/partition flow is very similar to the device annotate/rewrite flow. For comparison:
+
+ | Feature | Device Planning | BYOC |
+ | --------------------- | -------------------------- | ----------------------------------------------- |
+ | Source of annotations | `on_device`, `device_copy` | `FTVMAnnotateTarget`, `MergeComposite`+patterns |
+ | Target of planning | DLDeviceType | Toolchain name |
+ | Propagation | Unification based | Ad-hoc |
+ | Relay support | Full | First-order, no ADTs |
+ | Delimiting | insert `device_copy` | insert `compiler_begin`, `compiler_end` |
+ | Multiple per expr | No | Yes (though always picks first) |
+ | Hoists into functions | No | Yes |
+ | Customized heuristics | No | No |
+
+ Taking the 'upper bound' of the two implementations seems ideal, especially to address issues 4 (limitation
+ of device planning) and 5 (limitation of BYOC) above.
+
+Our proposal is:
+1. We introduce a new FFI-friendly class to represent a *S*torage or *E*xecution *Scope*:
+
+ ```
+ class SEScope {
+ DLDeviceType device_type;
+ int virtual_device_id;
Review comment:
i think this should be a String name which makes sense to the user. Doing this is helpful for a couple other reasons besides the compilation UI:
- In generated source code, it's possible to refer to the device by name. In particular, the embedded C API would like to have this for the conglomerate tvm_device_t struct.
- In systems with multiple e.g. CPUs, using an index here then implies some ordering (e.g. littlest CPU to biggest). It's better to make the assignment of ID to CPU capability more explicit
Finally, using a name would simplify the heterogeneous Target.
However, this is a bit of a lift. I do feel strongly we should get to this world. If it's not something that makes sense to do now, we could also revisit after or concurrent with USMP.
##########
File path: rfcs/0038-unified-device-target-and-memory-scope-planning.md
##########
@@ -0,0 +1,244 @@
+- Feature Name: unified-target-device-and-memory-scope-planning
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0038](https://github.com/apache/tvm-rfcs/pull/0038)
+- GitHub Issue: [apache/tvm#9327](https://github.com/apache/tvm/issues/9327)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated
+on more than one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain `on_device` annotations which specify that a sub-expression's result
+ should reside on a device with a given `DLDeviceType` (`kDLCPU`, `kDLCUDA`, etc).
+2. The `PlanDevices` pass uses those annotations to decide the unique device for every Relay
+ sub-expression, including every primitive operator call. Sub-expressions which are unconstrained
+ are assigned to the 'default' device. The pass then inserts `device_copy` operators whenever data
+ needs to cross device boundaries.
+3. The user must also supply a list of `Target` objects. The compiler uses that list to build
+ a `TargetMap` from `DLDeviceType` to `Target`.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile
+ ('lower') that primitive for that device. The `Target` to use for that compilation is found from
+ the `TargetMap` by the `LowerTEPass`.
+
+For the BYOC flow things are quite different:
+1. Operators may be annotated with an `FTVMAnnotateTarget` function for a particular
+ `target.<name>`. Here `<name>` serves only to distinguish possible BYOC toolchain names and is
+ currently not connected to the `Target` machinery in any way. The function should return true if
+ the given expression could be compiled for toolchain `<name>`. (However there are currently no
+ examples of this annotation in-tree.)
+2. The `MergeComposite` pass can be used to assign a `"Composite"` attribute to Relay functions
+ which have been hoisted out of a larger expression based on a fusion pattern. The attribute can
+ have any value of the form `"some.arbitrary.prefix.<name>"`. Again, this indicates the function
+ could be compiled for toolchain `<name>`. (The EthosU compilation flow illustrates this approach
+ in-tree.)
+3. The `AnnotateTarget` pass looks for the annotations from (1) and (2) to decide the unique
+ toolchain name for every Relay sub-expression which should go via a BYOC path. The transitions in
+ to and out of those sub-expressions are marked with `compiler_begin` and `compiler_end`
+ annotations.
+4. The `PartitionGraph` pass hoists sub-expressions delimited by `compiler_begin` and `compiler_end`
Review comment:
or is this the part where we translate the former thing to the hierarchical representation, and this is just how the implementation happens to be now? maybe @jroesch can comment here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] manupa-arm commented on pull request #38: [RFC] Improved multi-target handling
Posted by GitBox <gi...@apache.org>.
manupa-arm commented on pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#issuecomment-932097824
Hi @mbs-octoml ,
I may have put a related comment here : https://github.com/apache/tvm/pull/8892#issuecomment-932020564
However, partitioning for devices of same kind is a step forward from unifying the BYOC and Device annotations.
Is this the RFC intended to cover these all ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] areusch commented on a change in pull request #38: [RFC] Improved multi-target handling
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#discussion_r721652382
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
Review comment:
5!
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly
+ devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer
+ uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an
+ arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators.
+ Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system.
+3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for
+ primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the
+ overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to
+ deal with these different conventions.
+4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations
+ should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied
+ `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that
+ `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets,
+ it is particularly important to ensure there's a single source of truth for available `Target`s.
+5. TVM also supports a 'BYOC' extension mechanism. This allows `"target.<target name>"` annotations to be placed on
+ primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target
+ annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph
+ pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this
+ machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate
+ concept from `Target` objects.
+
+In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope
+is for this RFC to clear the way to taking on 5 in the future.
+
+Our proposal is:
+1. Extend `Target` to have a `DLDeviceType` attribute.
+2. Allow `Target` objects to be registered under a globally unique target label. Registration may be 'static' (ie
+ built into the TVM compiler via another REGISTER macro) and 'dynamic' (ie injected for a particular run of the
+ compiler, eg as part of `tvmc` command line processing). (This machinery should be reconciled with the existing
+ CUDA-specific target registration map.)
+3. Change the "on_device" call attributes to use a string instead of an integers (ie `DLDeviceType`). The string
+ can be of the form `<target label>` or `<target label>:<device id>`. The former simply implies a device id of 0.
+4. Rework device planning to use a pair of `Target` and 'device id' instead of `DLDeviceType`:
+ ```
+ class TargetDevice {
+ public:
+ Target target;
+ int device_id;
+ }
+ ```
+ (We could also use a `Device` and accept the redundant `DLDeviceType` specification.) It is trivial
+ to go from an "on_device" label to a `TargetDevice` and back using the global `Target` registry.
+5. Remove all uses of `TargetMap`. For example, in `LowerTEPass` we simply use the `TargetDevice` associated with
Review comment:
do you propose any replacement in case we do need a map-like struct? `Map<target_label, Target>`?
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly
+ devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer
+ uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an
+ arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators.
+ Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system.
+3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for
+ primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the
+ overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to
+ deal with these different conventions.
+4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations
+ should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied
+ `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that
+ `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets,
+ it is particularly important to ensure there's a single source of truth for available `Target`s.
+5. TVM also supports a 'BYOC' extension mechanism. This allows `"target.<target name>"` annotations to be placed on
+ primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target
+ annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph
+ pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this
+ machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate
+ concept from `Target` objects.
+
+In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope
+is for this RFC to clear the way to taking on 5 in the future.
+
+Our proposal is:
+1. Extend `Target` to have a `DLDeviceType` attribute.
Review comment:
as i'm reading this first time: curious how this will evolve when we take on 5 above.
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly
+ devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer
+ uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an
+ arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators.
+ Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system.
+3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for
+ primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the
+ overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to
+ deal with these different conventions.
+4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations
+ should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied
+ `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that
+ `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets,
+ it is particularly important to ensure there's a single source of truth for available `Target`s.
+5. TVM also supports a 'BYOC' extension mechanism. This allows `"target.<target name>"` annotations to be placed on
+ primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target
+ annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph
+ pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this
+ machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate
+ concept from `Target` objects.
+
+In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope
+is for this RFC to clear the way to taking on 5 in the future.
+
+Our proposal is:
+1. Extend `Target` to have a `DLDeviceType` attribute.
+2. Allow `Target` objects to be registered under a globally unique target label. Registration may be 'static' (ie
+ built into the TVM compiler via another REGISTER macro) and 'dynamic' (ie injected for a particular run of the
+ compiler, eg as part of `tvmc` command line processing). (This machinery should be reconciled with the existing
+ CUDA-specific target registration map.)
+3. Change the "on_device" call attributes to use a string instead of an integers (ie `DLDeviceType`). The string
+ can be of the form `<target label>` or `<target label>:<device id>`. The former simply implies a device id of 0.
+4. Rework device planning to use a pair of `Target` and 'device id' instead of `DLDeviceType`:
+ ```
+ class TargetDevice {
+ public:
+ Target target;
+ int device_id;
+ }
+ ```
+ (We could also use a `Device` and accept the redundant `DLDeviceType` specification.) It is trivial
+ to go from an "on_device" label to a `TargetDevice` and back using the global `Target` registry.
+5. Remove all uses of `TargetMap`. For example, in `LowerTEPass` we simply use the `TargetDevice` associated with
+ every primitive operator call already found by device planning.
+6. Bind two `TargetDevice`s as attributes on every `IRModule`:
+ - The default for primitive operators not otherwise constrained by "on_device" annotations.
Review comment:
and it seems like we could call this `target_executor` e.g. describing the Target where the executor should run. we should formally notate that now that we have an AOT flow.
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly
+ devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer
+ uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an
+ arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators.
+ Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system.
+3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for
+ primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the
+ overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to
+ deal with these different conventions.
+4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations
+ should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied
+ `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that
+ `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets,
+ it is particularly important to ensure there's a single source of truth for available `Target`s.
+5. TVM also supports a 'BYOC' extension mechanism. This allows `"target.<target name>"` annotations to be placed on
+ primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target
+ annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph
+ pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this
+ machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate
+ concept from `Target` objects.
+
+In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope
+is for this RFC to clear the way to taking on 5 in the future.
+
+Our proposal is:
+1. Extend `Target` to have a `DLDeviceType` attribute.
+2. Allow `Target` objects to be registered under a globally unique target label. Registration may be 'static' (ie
Review comment:
i'm curious about static vs dynamic Targets. is this the distinction we want to point out here, or do we just want to say there are Target aliases which may be mapped to pre-canned Targets?
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
+1. TVM is being targeted to environments with multiple CPUs (eg Arm 'Big.LITTLE') and multiple tensor-friendly
+ devices (eg a GPU as well as an accelerator such as Arm 'Ethos-U'). This means a `DLDeviceType` no longer
+ uniquely determines a `Target`.
+2. Though TVM's `Device` abstraction (an alias for `dlpack`'s `DLDevice`) is a pair of a `DLDeviceType` and an
+ arbitrary 'device id', TVM does not consistently plumb the device id through annotations, passes and operators.
+ Thus currently we cannot use 'device id' to distinguish, eg, two CPUs in the same system.
+3. The codebase still uses an older `target` and `target_host` convention for distinguishing the main `Target` for
+ primitive operators from the `Target` for residual tensor computation, shape computation, and (for AOT) the
+ overall Relay control-flow. There's a fair bit of 'target normalization' scattered throughout the codebase to
+ deal with these different conventions.
+4. `Target`s are often manufactured on-the-fly (eg to represent the default 'CPU' target on which shape computations
+ should be hosted). However there's no guarantee those default `Target`s will match up with the user-supplied
+ `Target`s, thus it's possible to end up with `"llvm"` and `"llvm -m ..."` `Targets` coexisting. Now that
+ `IRModule` uses `Target` objects themselves to distinguish which `PrimFunc`s are intended for which targets,
+ it is particularly important to ensure there's a single source of truth for available `Target`s.
+5. TVM also supports a 'BYOC' extension mechanism. This allows `"target.<target name>"` annotations to be placed on
+ primitive operations to indicate they should possibly be compiled with the matching BYOC toolchain. A target
+ annotation pass uses those annotations to decide on a target name for every Relay sub-expression. A partition graph
+ pass then inserts function call boundaries whenever execution needs to cross target boundaries. However this
+ machinery is separate from and incompatible with the "on_device" mechanism, and 'target names' are a separate
+ concept from `Target` objects.
+
+In this RFC we tackle problems 1-4. We won't directly take on 5 since it involves more moving parts, but our hope
+is for this RFC to clear the way to taking on 5 in the future.
+
+Our proposal is:
+1. Extend `Target` to have a `DLDeviceType` attribute.
+2. Allow `Target` objects to be registered under a globally unique target label. Registration may be 'static' (ie
+ built into the TVM compiler via another REGISTER macro) and 'dynamic' (ie injected for a particular run of the
+ compiler, eg as part of `tvmc` command line processing). (This machinery should be reconciled with the existing
+ CUDA-specific target registration map.)
+3. Change the "on_device" call attributes to use a string instead of an integers (ie `DLDeviceType`). The string
+ can be of the form `<target label>` or `<target label>:<device id>`. The former simply implies a device id of 0.
+4. Rework device planning to use a pair of `Target` and 'device id' instead of `DLDeviceType`:
+ ```
+ class TargetDevice {
+ public:
+ Target target;
+ int device_id;
+ }
+ ```
+ (We could also use a `Device` and accept the redundant `DLDeviceType` specification.) It is trivial
+ to go from an "on_device" label to a `TargetDevice` and back using the global `Target` registry.
+5. Remove all uses of `TargetMap`. For example, in `LowerTEPass` we simply use the `TargetDevice` associated with
+ every primitive operator call already found by device planning.
+6. Bind two `TargetDevice`s as attributes on every `IRModule`:
+ - The default for primitive operators not otherwise constrained by "on_device" annotations.
+ - The default for non primitive operators, such as Relay control flow and shape computation.
+7. We remove the various copies of target/target_host reconciliation, `TargetMap`
+ construction and 'default/fallback' device calculation from the codebase.
+
+This proposal tackles the original problems:
+1. There's now no ambiguity about `Targets` since we propagate them from the global registry directly.
+2. We support device ids.
Review comment:
it would be great to clarify between device IDs or target_label. Device ID is very much a runtime thing and we can't always control which e.g. GPU gets mapped to 0 or 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml commented on pull request #38: [RFC] Improved multi-target handling
Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#issuecomment-939103388
More notes to self:
- You can get the device_type for a Target from target_kind->device_type. Somehow I missed that very obvious fact.
- Eric has a very nice write-up explaining devices vs targets vs device api at docs/dev/device_target_interactions.rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] tqchen commented on pull request #38: [RFC] Improved multi-target handling
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#issuecomment-939150864
cc @ zxybazh since you authored the Target system, cc @zhiics @comaniac as it is related to BYOC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] mbs-octoml commented on a change in pull request #38: [RFC] Unified device/target/memory scope planning
Posted by GitBox <gi...@apache.org>.
mbs-octoml commented on a change in pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#discussion_r734003087
##########
File path: rfcs/00xx-improved-multi-target-handling.md
##########
@@ -0,0 +1,176 @@
+- Feature Name: improved-multi-target-handling
+- Start Date: 2021-09-20
+- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000)
+- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000)
+
+# Summary
+[summary]: #summary
+
+TVM supports 'hetrogeneous' execution, whereby primitive operators may be (sequentially) evaluated on more than
+one device (GPU, CPU, accelerator, etc). For the non-BYOC flow this works as follows:
+1. Relay programs may contain "on_device" annotations which specify that a sub-expressions's result should
+ reside on a device with a given `DLDeviceType` (kDLCPU, kDLCUDA, etc).
+2. The device planning pass uses those annotations to decide on the unique device for every Relay sub-expression,
+ including every primitive operator call. Sub-expressions which are unconstrained are assigned to the 'default'
+ device. The pass then inserts "device_copy" operators whenever tensors need to cross device boundaries.
+3. The user/driver must also supply a list of `Target` objects. The compiler uses that list to build a `TargetMap`
+ from `DLDeviceType` to `Target` for all of those objects.
+4. Each call to a primitive operator for a particular `DLDeviceType` signals we need to compile ('lower') that
+ primitive for that device. The `Target` to use for that compilation is found from the `TargetMap`.
+
+This approach has 5 problems:
Review comment:
Now 6. Progress!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] comaniac commented on pull request #38: [RFC] Unified device/target/memory scope planning
Posted by GitBox <gi...@apache.org>.
comaniac commented on pull request #38:
URL: https://github.com/apache/tvm-rfcs/pull/38#issuecomment-955085747
For the proposed BYOC flow (i.e., `MergeAndAnnotate`/`AnnotateSEScopes`/`PlanDevices`/`PartitionBySEScope`), it doesn't clear to me that whether the developer programming model will change or not. Specifically, could we still use the current approaches (i.e., op-based annotation and pattern-based annotation)? Also how the approach that has a custom Relay pass to annotate `compiler_begin`/`compiler_end` would change? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org