You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tvm.apache.org by Christopher Sidebottom via Apache TVM Discuss <no...@discuss.tvm.ai> on 2021/11/01 14:27:31 UTC
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Inspired by the work of @mbs-octoml, I give you a new RFC for CompilationConfig!
# Summary
[summary]: #summary
This RFC supersedes [Migrating IRModules to Attributes](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0029-migrating-to-irmodule-attributes.md) by replacing the various attributes associated with the `Runtime` and `Executor` with a single `CompilationConfig` object that encapsulates the configuration of the compiler for a given `IRModule`. By collecting this together, it introduces on object which can be coupled to the `IRModule` and used through-out compilation with guarantees about the properties of the configuration.
# Motivation
[motivation]: #motivation
## Argument Duplication
When implementing [Migrating IRModules to Attributes](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0029-migrating-to-irmodule-attributes.md), it became clear that the arguments were getting duplicated in many places across the codebase and never collated. Such as the following `tvmc` CLI:
```
tvmc --target=c --executor=aot --runtime=crt
```
Which populates the arguments `executor`, `runtime` and `target` all the way into the compilation flow, rather than collating them and passing the pre-processed representation. This introduces places we can make errors due to missing one of the three and always having to replicate the signature.
## Single Point of Setup
Futher to this, many areas of the code required access to the compiler configurations non-optionally, this introduces uncertainty in later passes which should be able to guarantee the compiler has been configured - thus making the dynamic attributes less ideal for this use-case:
```cpp
// Hopefully this is set!
ir_mod->GetAttr<Executor>(kExecutor).value()->name;
```
As can be seen by [adding SEScope](https://github.com/apache/tvm/pull/9313), there's also need for a single place to collect and define configuration rather than cause duplication of effort, with multiple calls to setup various parts of the configuration, such as `Target` environment which should be known when the compilation flow begins:
```cpp
CheckAndUpdateHostConsistency(&target, &target_host);
```
By moving these to a single property of the `IRModule` which is required to exist before entry into the compiler flow, this reduces both the cognitive overhead of accessing the configuration and provides a place to encapsulate the logic of setup early in the compilation flow.
This is a fixed set of configuration alongside the existing `PassContext::Global()` to solidify the configuration requirements of the TVM Compiler.
# Guide-level explanation
[guide-level-explanation]: #guide-level-explanation
## User API
A user creates and passes a configuration to `relay.build(mod, config)`, leading to something similar to:
```python
ir_mod = IRModule.from_expr(function)
config = CompilationConfig(
target_host=target,
targets=[target],
executor=Executor("aot", {"interface-api": "c", "unpacked-api": True}),
runtime=Runtime("crt", {"system-lib": True})
)
relay.build(ir_mod, config)
```
To easily create a default `CompilationConfig` from a single `Target`, the `from_target` option is provided - this can be used to add syntactic ease in `relay.build`:
```python
def relay.build(ir_mod, config, ...):
if isinstance(config, Target):
config = CompilationConfig.from_target(config)
```
The C++ API is then amended to require configuration:
```cpp
/*!
* \brief Build relay IRModule
*
* \param mod Relay IRModule
* \param config Compilation Config for this compiler run
* \param mod_name Name of the module
*/
void Build(IRModule mod, const CompilationConfig& config, const String mod_name)
```
This means a user can continue to easily create and pass an `IRModule` to `relay.build` as an opaque object whilst providing the configuration alongside.
## Developer API
The developer facing API for the `IRModule` changes for internal passes to easily access the configuration:
```cpp
ir_mod->GetConfig()->GetExecutor()
```
And functionality related to inspecting this configuration can be added to `CompilationConfig` which can see all available configuration properties:
```cpp
ir_mod->GetConfig()->ShouldLinkParams()
```
Importantly, for ad-hoc information passed alongside with the `IRModule`, attributes continue to be available:
```cpp
WithAttr<String>(ir_mod, "woof", "woof");
```
# Reference-level explanation
[reference-level-explanation]: #reference-level-explanation
## IRModule Configuration
To incorporate this into the `IRModule`, a property of type `CompilationConfig` will be added and exposed via a function `GetConfig()` in line with [the Google C++ guidelines](https://google.github.io/styleguide/cppguide.html) except for being a public property for TVMs internal node structures:
```cpp
class IRModuleNode {
CompilationConfig config;
const CompilationConfig& GetConfig() {
return config;
}
}
```
The actual `CompilationConfig` class will represent the current and future shape of configuration, such as `Executor` and `Runtime`, alongside `Target` and `Target` host (example illustrative not carved into stone):
```cpp
class CompilationConfigNode {
Target target_host;
Array<Target> targets;
Runtime runtime;
Executor executor;
}
```
From Python the module will be passed into C++ with the `CompilationConfig` to start the compilation flow:
```python
def relay.build(ir_mod, config: Union[CompilationConfig, Target], ...):
if isinstance(config, Target):
config = CompilationConfig.from_target(config)
mod["build"](ir_mod, config)
```
Which is then connected within C++:
```cpp
void Build(IRModule mod, const CompilationConfig& config, const String mod_name) {
mod->config = config;
...
}
```
When creating `IRModule` -> `IRModule` passes within the compiler this property should now be guaranteed to exist in the main Relay flow.
## Non-Relay Flows
There are a number of ways of accessing the internals of TVM which have been exposed to the user, for these to continue to function the `CompilationConfig` will be settable from Python such that:
```python
ir_mod = ir_mod.with_configuration(config)
```
The above will attach the configuration within the alternative pathways, it is the opinion of the author that these should be eventually deprecated in favour of a single standard compilation path.
## Compilation Configuration Creators
Within `python/tvm/target/target.py` there are a number of functions which returned `Target`s with configuration built in, such as:
https://github.com/apache/tvm/blob/e8077439efbe021d73cf27018cd83653f964255f/python/tvm/target/target.py#L305-L328
These are user-facing configurations, and will be ported to `tvm/target/config.py` to match the `include/tvm/target/compilation_config.h` and provide the same logic but returning a `CompilationConfig`:
```python
def micro(model="unknown", options=None, executor="graph"):
opts = _merge_opts(
MICRO_SUPPORTED_MODELS[model] + ["-runtime=c", f"-model={model}"],
options,
)
target = Target(" ".join(["c"] + opts))
return CompilationConfig(
target_host=[target],
targets=[target],
executor=Executor(executor),
runtime=Runtime("crt", {
"system-lib": executor == "graph"
})
)
```
# Drawbacks
[drawbacks]: #drawbacks
- This is more information on the `IRModule` alongside existing attributes and pass information
- The entry contract for `relay.build` and the TVM Compiler changes to require configuration of the initial `IRModule`
# Rationale and alternatives
[rationale-and-alternatives]: #rationale-and-alternatives
Continue to use `PassContext::Global()` and general purpose attributes for fixed compilation configuration, using the general purpose options here hides the details of the compilation configuration and makes it harder for us to provide a robust representation of none-`Optional` configuration. If all things are considered `Optional` then they have to be considered as none-configured and requiring multiple passes of reconfiguration to provide safety to the users.
# Prior art
[prior-art]: #prior-art
- [Migrating IRModules to Attributes](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0029-migrating-to-irmodule-attributes.md) split the settings from `Target` into `Executor` and `Runtime` attributes
- `CompilationConfig` was introduced in [Add SEScope PR #9313](https://github.com/apache/tvm/pull/9313#issuecomment-955732036)
- `PassContext` already exists in TVM as a way of passing general configuration
# Unresolved questions
[unresolved-questions]: #unresolved-questions
- Do we want to fully deprecate passing a `target` to `relay.build`? The assumption is made that we don't want to fully deprecate it and a sugar has been added above for populating a `CompilationConfig` straight from a `Target`
- Similarly, do we want to provide a similar helper for `target` and `target_host`:
```python
def relay.build(ir_mod, config, target_host=None):
if isinstance(config, Target):
config = CompilationConfig.from_target(config, target_host=target_host)
```
- What is the future for `IRModule` attributes and `PassContext` - this RFC aims to provide one piece of concrete configuration which we know can be fixed at the start of compilation but there should be a default choice for ad-hoc configuration within the compilation flow.
# Future possibilities
[future-possibilities]: #future-possibilities
By codifying the properties of the compilation flow in `CompilationConfig` and providing `IRModule` attributes we may be able to deprecate `PassContext::Global()` and only use the passed state in TVM.
@areusch @manupa-arm @jroesch @tqchen
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/1) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/c291481dae0f3b187d191b56089daddd031577e452a2a3568b1ad586f99a425f).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by tqchen via Apache TVM Discuss <no...@discuss.tvm.ai>.
https://discuss.tvm.apache.org/t/rfc-composite-target/7744
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/7) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/134e69e6583759a2bed230a71d1a5d055827b3e80879250c6a71234171474c71).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by tqchen via Apache TVM Discuss <no...@discuss.tvm.ai>.
To elaborate on C2, while it is desirable and recommended to have consolidated runtime, executor choice when possible. Naturally there are cases that would requires a bit of generalization. The multi-machine case is one example.
There are also other examples that can appear on a single SoC. Consider the following scenario, where there is an accelerator that comes with a CPU-like co-processor as controller.
```bash
- host: arm
- runtime: vm
- vdevice0: accelerator-with-coprocessor
- host: risc-v
- runtime: graph
- device: my-accelerator
```
In this case, the host is a ARM chip that drives the overall computation(say through VM). The co-processor, however, also comes with its own controller, that is able to execute a sub-graph of computation, which in turn dispatches to my-accelerator. As a result, we will need to compile a tvm runtime(that may be different from the host) one, and use that to drive the graph computation on the co-processor.
Back to the compilation path. I agree that it is important to build a standard pipeline. I would also like to note that we need to design to be compatible of emerging needs. Allowing target specification to be recursive, while validating them, would help the ecosystem to develop these capabilities. Additionally, some of the needs can appear now, for example, we could see a need to have a more flexible VM runtime that drives GPU computation, while offloading subgraph to cuda-graph(more efficient and less flexible).
Finally to build on @Mousius 's point. Allowing target to be recursive does not preclude structure or naming. Targets have kinds and schemas that attached to each kind. Further validation can also be done throughout the process. So instead of
```
(CompilationConfig)
-> (Target-CUDA), (Target-X86)
-> (Executor)
-> (Runtime)
```
We would get
```
(Target-Kind=Hetro-Exec)
-> (Target-Kind=CUDA), (Target-Kind=X86)
-> (Executor)
-> (Runtime)
```
From the UX's pov, we do not need to force user to pass in such compositional ones(that is complicated) if they only care about single device execution (and canonicalize internally).
Even better you will be able to leverage the tagging features at different level, so user can just pass in
```python
build(mod, target="my-hetro-exec-platform0")
```
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/14) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/fdd5931911f5cde301da2e3daa5b83320acc2535fac0b7dd57f607389f78e971).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Christopher Sidebottom via Apache TVM Discuss <no...@discuss.tvm.ai>.
Oh wow, I've been away for a few days and really appreciate the amount of discussion that's arrived :smile_cat: Thanks @mbs-octoml, @zxybazh, @tqchen, @comaniac, @junrushao1994 and @manupa-arm!
Firstly let's address a few specifics which may help narrow the discussion slightly:
[quote="junrushao1994, post:10, topic:11372"]
Last, there are cases where multiple executors working together. For example, if we want to offload some fragments to TensorRT executor, some to CUDA graph, while keep the rest in the VM, then the Relay function could potentially be partitioned into 3 Relay functions that targets to different executors. With composite target, we are able to attach different executors in the Target object in a more explicit way.
[/quote]
There's an unfortunate overloading of the terms `Executor` and `Runtime` which is the inherent risk with a diverse heterogeneous compiler :smile_cat:. In this RFC, let's define the `Executor` and `Runtime` as specific to the TVMs `Executor` and `Runtime` rather than the implementation of a `Target`. How a `Target` gets generated and linked is outside the immediate scope of the TVM `Runtime` and `Executor` as they're designed to invoke the generated `Target` code.
[quote="mbs-octoml, post:3, topic:11372"]
* I think you want your tvmc cmdline flags to be interpreted as rewrites on CompilerConfig in some uniform way, right?
[/quote]
[quote="junrushao1994, post:10, topic:11372"]
The RFC text doesn’t mention how it could improve the UX in TVMC command line. However, I would argue that the UX could be improved simply with target tags. For example, on CUDA GPUs, our target tag system supports creating CUDA targets with a single short string.
[/quote]
Thanks @mbs-octoml, I missed some Prior Art here! In `tvmc`, we have the concept of a configuration as defined in [Command Line Configuration Files](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0030-tvmc-comand-line-configuration-files.md). `CompilationConfig` would allow this to be a standard way of defining such as configuration with `Target`s within it - this meets the needs of the Target tagging which @junrushao1994 and @tqchen are discussing by instead wrapping them into a `CompilationConfig` that represents the system. Within the [Command Line Configuration Files RFC](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0030-tvmc-comand-line-configuration-files.md), it defines the `<TYPE>` and indicates the use of `--config` for cloud instances. The terminology would shift from a tagged `Target` to a `CompilationConfig` here to represent they exist at two different levels of the hierarchy?
[quote="tqchen, post:12, topic:11372"]
* I do not think there is a strong contention on C1, the main point is that the target can be recursive. So a target like the follows is totally OK.
[/quote]
As defined in [Migrating Target Attributes to IRModule](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0029-migrating-to-irmodule-attributes.md), splitting the TVM concepts of `Target`, `Runtime` and `Executor` means we can more clearly see what is most relevant to a specific `Target`. Which means that call-site annotations for `Target` are limited to only options that are relevant to a specific `Target` rather than to an `IRModule`. By virtue of working on this RFCs implementation, although we should still land the implementation agreed in the RFC, it does illustrate how we can better manage the representation of this configuration internally to TVM.
One reason not to motivate this as purely a `tvmc` concern is that `tvmc` is the CLI interface to TVM, if a user attempts to use `tvmc` and then moves to a Python script they should not be re-learning the interface to TVM.
[quote="tqchen, post:5, topic:11372"]
G1: Ability to enable incremental customization (via python API), attach constraints(such as BYOC) and then send back to the build function for further lowering.
[/quote]
This sounds sensible, and also a feature of `CompilationConfig` is the ability to specify the complete picture of the system which TVM is being built for including all `Target`s which can be used by all `Pass`es. Specific annotations of storage and execution make sense to be defined at call-sites within the IR rather than at the top level with the `IRModule` - what `CompilationConfig` provides is a frame of reference to do those annotations and pick from a variety of `Target`s and `Device`s which the `IRModule` is constrained to. As we continue with [`Target` registered compiler flow customisation](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0010-target-registered-compiler-flow-customisation.md), annotating a call-site with a `Target` will become standardised with the BYOC flow whether partitioned or otherwise to match the expectation you described with partitioned `Target`s.
[quote="zxybazh, post:4, topic:11372"]
The design of targets list and target host seems really similar to a composite target, can we actually use a single composite target (where each target can be a composite target) with executor and runtime field as compilation config?
[/quote]
This doesn't rule out the possibility of using a composite `Target` as a `Target` in the `targets` list as we're not redefining how that works here - rather defining a bounding box for system level configuration within TVM.
[quote="zxybazh, post:4, topic:11372"]
`CheckAndUpdateHostConsistency` is designed to make sure target.host and target host are consistent along the way. If we have a better machanism to do that since we have target host as a field in compilation configuration, we may further reconsider the usage of target inside of compilation flow.
[/quote]
The end state for this configuration update would be to run a single pass over the `CompilationConfig` early on to ensure the internal state was correct using `CheckAndUpdateHostConsistency` which guarantees that subsequent `Pass`es such as device or memory planning are safe making assumptions about the state of the used `Target`s. Hopefully that clarifies it's less of a replacement, but more of a consolidation of the logic to early in the compilation flow if these checks are still required :smile_cat: We'd still need to have `Target` annotations within the IR and that `Target` will therefore have to be stable during compilation.
## Where we're at
Going over this thread a few times, the discussion revolves around:
M0. Split the CompilationConfig from Target
```
(CompilationConfig)
-> (Target), (Target), (Target)
-> (Executor)
-> (Runtime)
```
M1. Recursively allowing Target to represent any system
```
(Tagged Target)
-> (Target), (Target), (Target)
-> (Executor)
-> (Runtime)
```
It is my opinion, and the motivation behind this RFC, that better defining the `CompilationConfig` would relieve cognitive load on the user and provide definitions which can be bounded easily. By continuing to use M1 the term `Target` is increasingly overloaded and difficult for both developers of TVM and more importantly users of TVM. This hierarchical terminology has prior art in large scale cloud frameworks, such as Kubernetes which uses different terminology for `Cluster`, `Deployment`, `Service`, `Pod` and `Container` which are all different levels of granularity of computing resources; the decision here is both a UX decision and a practical separation of concerns for both users and developers of Kubernetes.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/13) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/245bfd5f645af4809aa78d336679fadf06da4095f7be61d0e044fb31bef619a2).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by tqchen via Apache TVM Discuss <no...@discuss.tvm.ai>.
Thanks @manupa-arm , building on what you said.
- I do not think there is a strong contention on C1, the main point is that the target can be recursive. So a target like the follows is totally OK.
```bash
- kind: hetro-exec
- runtime : crt
- executor: vm
- devices: [ se_scope0, se_scope1 ]
```
So the argument is not about where/or how these field should be structured in a recursive data structure. Something that looks like a CompilationOption is OK from my pov. But the suggestion is that we make that one kind of (recursive) target, as from UX's pov it can be seen that way.
- I want to add C3: The ability to leverage tagging in target and improve the overall user experience is a very important factor.
I am going to discuss C2 on a separate post since that worths more examples.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/12) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/84633aded8b0425112ceeadeb2830f9fdcea2ccca31e5294b066aaf806ea4358).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Manupa Karunaratne via Apache TVM Discuss <no...@discuss.tvm.ai>.
Thanks for the interesting discussion.
@tqchen @junrushao1994 ,
In terms of the definition of the target, I see two categories of arguments presented here :
C1 : The executor, runtime, <someother option that is not exclusively attributable to the target> should belong to the target -- even if means duplication.
C2 : The targets should be hierarchical and recursive
For C1, I would rather use this argument to make runtime and executor an attribute of the target rather than to support calling an Array of Targets, another target. I can see this being true in the following scenario (as pointed out by @tqchen), if its a scenario we want to target for.
The following scenario is motivated by the fact it is economical run a single model inference across multiple machines considering the data transfer costs of intermediary tensors of the model. Just want to make sure if this is something the community considers as a compilation scenario that TVM should aim for.
[quote="tqchen, post:8, topic:11372"]
As the field evolves, the concept of “target” can change further. Right now we are talking about a single SoC with multiple devices. What if we develop an interest in deploying onto the following distributed environment.
```
- machine0:
- host: x86
- vdevice0: cuda
- machine1:
- host: arm
- vdevice0: vulkan
```
[/quote]
For C2,
The examples presented so far does not seem to go beyond mostly a flat Array of Targets. Maybe in the multiple machine scenario, an alternative could have been a Array of CompilationConfig (or whatever we decide to call it). However, This would not be viable if we have recursive targets (where the recursion depth > 1)
Do you guys see a likely scenario in which we will have a Composite Target that is composed of Composite Targets of Composite Targets ? (i.e. where we cant express the target we want to compile to as a Array of Targets coupled with a host target -- I believe host target differs only in the multiple machine scenario),
If that is the case, how would TVM establish codegen path divergence (partitioning at different levels of IR) to such an hierarchical target ?
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/11) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/d92ba0aa7cb8ee4a059311ccd29b7b6976d7004992c185a3fd2e80a72ff768c5).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by tqchen via Apache TVM Discuss <no...@discuss.tvm.ai>.
Thanks for the discussions. To begin with, I am not that attached to the
particular choice of name. We can for example, decide to introduce another target kind
("hetero-target", "myawesome-target", "platform", "CompilationOption").
I think our discussion boils down to the following quenstion
## What can be called a "Target" in tvm
Intuitively, to many users, target refers to the "target platform" or environment
that they want to run the program on. In a typical clang target triple, the
following elements can be part of a target:
- ISA (x86, arm, riscv)
- runtime library (musl, libc)
- operation system env (windows, linux)
- vendor
Of course in most of the settings here target refers to a single device, usually
with a single codegen path. These are targets at the leaf level.
However, as we start to build compilers for ML. The "target" in users' mind is different.
For example, I want to run my program as fast as possible on `aws/c4.4xlarge`, or `nvidia/jetson-nano`.
Some of these "targets" already involves multiple codegen path(host code and device code).
When we start to involve graph or vm for the high level program driver, the vm/graph/aot choice
is another codegen path on the driving path of the program.
As the field evolves, the concept of "target" can change further. Right now we are talking about
a single SoC with multiple devices. What if we are interested in deploying onto the following
distributed environment.
```
- machine0:
- host: x86
- vdevice0: cuda
- machine1:
- host: arm
- vdevice0: vulkan
```
We might also interested in the following byoc customization where we offload part of
the computation to `byoc-myawesome-cuda` strategy, which needs a self-contained
specification of host and library targets that makes use of cuda-graph runtime.
We want to embed it in a vm runtime that invoke the byoc-myawesome-cuda as an opaque function.
```
- host: x86
- vdevice0: byoc-myawesome-cuda
- host: x86
- runtime: cuda-graph
- vdevice0: cuda
- library: tensor-rt
- vdevice1: cuda
- runtime: vm
```
Can we call the above descriptions as "target"? From a UX's perspective they certainly
can be called target. Since from user's perspective it is a specification of
"target environtment". In the context of machine learning they certainly can usually
go beyond a single codegen path.
Another thing to note here is that some of these examples requires a level of
compositionality that goes beyond one level. In the context of multi-machine setting,
the setting per machine roughly maps to CompilationOption being used here. Similarly,
in the case of `byoc-myawesome-cuda`, vdevice0 itself would benefit from its
own runtime specification.
Another concept(another target kind) is needed to introduce another concept in order to
support the top-level composition.
## UX Benefit of a Target -- Tagging
Besides the benefit of the compositionality, one major UX benefit of target is the ability to tag.
It can be really complicated to manually specify a compositional compilatin option.
In most cases, we want users to directly leverage pre-built tags. For example, build for
nvidia/jetson-nano:cuda, build for `aws/c4.4xlarge`, build for `arm/soc-name:aot`(that directly implies unpacked_api).
These tags create short hands for us to setup the compositional configurations.
The ability to let build function takes in tags that quickly maps to both codegen, runtime,
and library configurations would greatly improve overall user experiences.
Making CompilationOption would allow us to reuse this feature effectively and recursively.
## Discussions
The main discussion point here is what is the scope of target.
As we can see that:
- A0: On one hand, we can say that the configuration strictly follows a two-level
structure. Where target is on the leaf, specifies a single codegen path. While we use
a separate name for the top-level compositions.
- A1: On the other hand, we can see the need of:
- More than two levels of compositions
- The UX need to reuse tagging mechanism and simplify users' inputs to the compiler.
From a two-level compositional view. Personally I think reusing Target for CompilationOption
is not strictly more complicated, modulo the right kind naming. While the needs in ML can
certainly go beyond that. Which makes me think going for target compositionality is not a bad idea.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/8) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/969166766f1ec0b4bd1cc5670480ce01237f7f722f09616c8be941f7417952a7).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Manupa Karunaratne via Apache TVM Discuss <no...@discuss.tvm.ai>.
Hi @tqchen and @zxybach,
cc : @mbaret
What is a Composite Target ?
TVM being a multi-target compiler, it would be a bit confusing to use a Array of Targets as another Composite Target -- I think its the terminology what is confusing here.
A composite target sounds like a target that codegen intimately in a single codegen path for different devices rather than a structure that is used by TVM to trigger different codegen flows. I think we all agree we need a way to communicate this Options/Flags through out the lowering but I personally would not be in favor of attaching this to a (Composite) target -- that result in overloading the term "Target".
[quote="tqchen, post:5, topic:11372"]
We get the benefit of being able to tag, and record target
[/quote]
I believe we can still do it as the target is part of the CompilationConfig
[quote="tqchen, post:5, topic:11372"]
CompilationOption can appear as a field of sub-target of something else. Imagine that we need to offload a subgraph to another customized compilation, which may needs its own specification of the heterogenous “targets”.
[/quote]
CompilationConfig is supposed to contain attributes that are not specific to a target. Thus, they would be still accesible in the IRModule. Wont they ?
[quote="tqchen, post:5, topic:11372"]
Same API argument(Target) for both graph level compilation and operator level compilation.
[/quote]
This could also be said with respect to CompilationConfig being available for both graph level compilation and operator level compilation -- just that "targets" are part of CompilationConfig.
In a summary what we need to discuss is :
* What is the best term to use to structure that holds information/flags that are target independent and hold the set of targets in the same time?
* Moreover, it would be great to reserve the term "Composite" target to a target that intimately codegen to multiple devices without divergence in the compilation pathway.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/6) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/38724dac9969b14422fddc5d4ef71ce2cae13feb51e53a7f59aea4cb48ebbc65).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by tqchen via Apache TVM Discuss <no...@discuss.tvm.ai>.
Thanks for the discussions. I think it is a good opportunty to discuss how can we flow target information through the compilation. Putting some fruits of thoughts
## How to flow target information through compilation
One of the main design goal that we want to move towards to is this ability to incrementally transform the code(some of the transformations may not be done in the official build pipeline). Take BYOC as an example, in the future we might invoke a custom pass that slices out a subgraph and generate a function that requires a specific target lowering(e.g. CUDA). The diagram below from TensorIR blitz course shows one example of such a flow:
In summary, there can be two goals:
- G0: Ability to config a single standard compilation path.
- G1: Ability to enable incremental customization (via python API), attach constraints(such as BYOC) and then send back to the build function for further lowering.
G0 is certainly sufficient for some of the usecases like tvmc. However, it is also important for us to take inspiration, and think more about making G1 as a first class citizen. A natural consequence of G1 is that we will need to preserve certain "target-constraint" information in IRModule(so previous transformations's decision are self-contained), either as attr of a function(e.g. this function have to be compiled in CUDA), or IRModule.
It would be great for us to collectively think about a way on how to standardize for G1 while still have the ability to support G0.
## CompilationConfig and Composite Target
Back to the CompilationConfig itself. I agree with @zxybazh that it looks quite like a special case of composite target and it is useful to discuss whether or not we can simply merge it as a structured Target.
Coming back to the definition of target, if we look at LLVM's target triple, `-arch-subarch-os-vendor-env-object format`. We can find that it also contains runtime choice information like ABI for the libc, OS type and so on. So one could argue that choices like `tvm_runtime` type, packed function API can be part of a composite target (although they do not need to be in the leaf "c").
The advantage of having a CompilationOption class:
- It is a structured class with explicit fields
The advantage of having making CompilationOption as a composite Target
- We still have structured fields with target configurations
- We get the benefit of being able to tag, and record target
- CompilationOption can appear as a field of sub-target of something else. Imagine that we need to offload a subgraph to another customized compilation, which may needs its own specification of the heterogenous "targets".
- Same API argument(Target) for both graph level compilation and operator level compilation.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/5) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/7fbee6ea4169030cc1223a098866ff7f58eed1719574566c08731607fd6e4c22).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Xiyou Zhou via Apache TVM Discuss <no...@discuss.tvm.ai>.
Thanks for the RFC. Generally I am in support of the change.
- Moving the config to IRModule could better avoid overloading the target.
- The design of targets list and target host seems really similar to a composite target, can we actually use a single composite target (where each target can be a composite target) with executor and runtime field as compilation config?
- `CheckAndUpdateHostConsistency` is designed to make sure target.host and target host are consistent along the way. If we have a better machanism to do that since we have target host as a field in compilation configuration, we may further reconsider the usage of target inside of compilation flow.
- On the target side, it's a breaking change but I would be optimistic on deprecation after a couple releases.
- Nit: personally I prefer to use target host as a field of target, or complication config, instead of directly have a argument name for that. `from_target` can also retrieve the target host from given target.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/4) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/2c7f328d4a623db865cb097605c63889666e9c71d64d95b46750e9f193931e18).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Junru Shao via Apache TVM Discuss <no...@discuss.tvm.ai>.
@Mousius I totally agree to make things hygiene, and believe folding things into Target is the correct and consistent approach.
First of all, the automation system solely relies on the target object to understand the code dispatching, hardware specs and runtime information. Without having the information in the Target object, the automation system won't be aware of the full picture. For example, if we switch executor from VM to TensorRT, the performance can be much different, and so if executor is not inside Target, then the automation system will be confused and learn a wrong objective.
Second, as the direction we are moving towards, the Target object is guiding our IRModule-to-IRModule transformation in lowering, and IRModule-to-Module in compilation. Wrapping with an extra layer seems to architecturally change our compilation pipeline, while alternatives do exist and both seem to be equivalently expressive.
Third, the practice folding all compilation-related information has been adopted consistently in TVM. For example, we may specify the libraries dispatched to via `cuda --libs=cudnn`. Similarly in LLVM, the target triple is designed in a consistent way, where we could specify libc and other environments.
Historically, fragmentation accumulates in TVM across layers. For example, we have different scheduling and auto scheduling system, slightly-but-not-identical and error-prone APIs for different executors, compilation workflow between relay, relay byoc and tvm, etc. Adding new top-level user-facing data structures, if alternative exists with the same expressiveness and UX, then I would say it would probably lead to more user confusion.
On the other hand, I totally agree and am aware that a graph-level compile involves the interaction of multiple parts, including device, host, runtime and executor. The main concern from me here is that we already have Target as a canonical spec formally, which is already able to express this structure without hurting UX.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/35) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/ecfdf6bd890535d6826a1faf303323fc10ab5cde3583aa530ddf4d0579590fa7).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Junru Shao via Apache TVM Discuss <no...@discuss.tvm.ai>.
Thank you @Mousius for the RFC! It's great to read about potential user experience issues of the current Target system, and happy to discuss about potential ways to improve it.
## Proposeds API in the RFC
`CompilationConfig`, as proposed in this RFC, aims to improve UX by wrapping a list of targets, runtime and execution information in an extra layer of abstraction.
The core API is demonstrated in the RFC as:
```python
config = CompilationConfig(
target_host=target,
targets=[target],
executor=Executor("aot", {"interface-api": "c", "unpacked-api": True}),
runtime=Runtime("crt", {"system-lib": True})
)
```
To improve the developer experience, a few other APIs are proposed along with the data structure:
```C++
CompilationConfigNode::GetExecutor();
CompilationConfigNode::ShouldLinkParams();
```
The compilation workflow changes from building with `Target` to building with `CompilationConfig`, as demonstrated below:
```C++
// The current API
void Build(IRModule mod, const Target& target, ...);
// The proposed API
void Build(IRModule mod, const CompilationConfig& config, ...);
```
## Existing Work
As proposed in the [target specification](https://discuss.tvm.apache.org/t/rfc-tvm-target-specification/6844) and [composite target](https://discuss.tvm.apache.org/t/rfc-composite-target/7744) RFCs, the existing effort converges to the following items.
First, `host` is folded into the Target object, and the `target_host` parameter in existing build APIs, in fact, are left for backward compatibility. The `CheckAndUpdateHostConsistency` API developed by @zxybazh, is only used for backward compatibility reasons. Right now, the canonical way to specify targets with customized host is as easy as:
```python
target = tvm.target.Target("cuda", host="llvm")
```
Second, in terms of multi-target and heterogeneous support, composite target is adopted as the current approach. Comparing composite target, which is target host plus a list of targets, with the proposed `CompilationConfig`, which is also target host plus a list of target, it seems very much following the same idea, while `CompilationConfig` is an extra layer of abstraction.
Third, canonical form of a Target is a JSON object, not a pain string. The target implementation already supports hierarchical parsing, e.g. target inside target inside array, etc. To support executor and runtime with attributes, we could extend the parser to support converting a JSON sub-object to an Executor/Runtime object, which is very much doable.
## Discussion on the RFC
Overall, the RFC brings a dramatic change to the compilation infrastructure. This effort enforces a new assumption that we only have a single executor and a single runtime. However, I could see clean alternatives with more expressiveness, less effort required, no breaking change, but achieve the same goal.
First, under our unified IR efforts, the compilation in TVM is heading towards `IRModule` to `runtime::Module` abstraction. The executor, to the best of my understanding, is a runtime object that executes some artifacts that some BaseFuncs lowers to. For example, VM executor interprets VM bytecode, AOT executor may run the binary directly. Right now, there are some leaky abstraction, but our goal should be aligned under the direction that we address those leaks instead of bringing in more.
Second, the proposed APIs seem to be possible to be implemented with straightforward helper functions under the current abstraction. To give a line-by-line example:
```C++
ir_mod->GetConfig() -> CompilationConfig; // proposed in the RFC
GetTarget(id_mod) -> Target; // alternative
ir_mod->GetExecutor() -> Executor; // proposed in the RFC
GetExecutor(id_mod) -> Executor; // alternative
ir_mod->GetConfig()->ShouldLinkParams() -> bool; // proposed in the RFC
ShouldLinkParams(id_mod) -> bool; // alternative
```
In short, using accessor pattern here doesn't bring in actual benefits, and can be replaced by simple helper functions.
Third, the RFC text doesn't mention how it could improve the UX in TVMC command line. However, I would argue that the UX could be improved simply with target tags. For example, on CUDA GPUs, our target tag system supports creating CUDA targets with a single short string:
```python
target = tvm.target.Target("nvidia/geforce-rtx-3070")
```
This carries all the information needed for a device, as long as we register them into our system, including compute version, shared memory size, local memory size, etc. This could perfectly solve the UX issue in TVMC by simply allowing target tags as arguments:
```python
tvmc --target "nvidia/geforce-rtx-3070"
```
Last, there are cases where multiple executors working together. For example, if we want to offload some fragments to TensorRT executor, some to CUDA graph, while keep the rest in the VM, then the Relay function could potentially be partitioned into 3 Relay functions that targets to different executors. With composite target, we are able to attach different executors in the Target object in a more explicit way.
## Conclusion
When designing the Target spec, it is intended to be considered as the synonym to `CompilationConfig`. I may not have all the context here and my understanding could be limited, but as heavily involved in the Target design, from my PoV, for now the benefit of the RFC seems to be limited to certain issues Target is already able to do. Happy to chat more!
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/10) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/d6662642434755de173b1551b0b1258f45d47352003055952c22fa043ee52323).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by "Cody H. Yu via Apache TVM Discuss" <no...@discuss.tvm.ai>.
I agree with @tqchen that improving composite targets could be more beneficial and general. We (with @junrushao1994 and @zhiics) previously attempted to improve the target system to allow more flexible attributes, such as a pass sequence / runtime / etc specifically for the target, which is very similar to what TQ illustrated and what this RFC proposed, but found that it's not an easy task due to the current target system implementation.
Meanwhile, the concept of compilation configuration has been used for some BYOC backends already, but they are currently relying on PassContext. For example, TensorRT codegen takes the configuration from PassContext during `relay.build`:
```python
mod, config = partition_for_tensorrt(mod, params)
target = "cuda"
with tvm.transform.PassContext(opt_level=3, config={'relay.ext.tensorrt.options': config}):
lib = relay.build(mod, target=target, params=params)
```
Although the `config` here is generated internally, I think this could still be a good driving example to see how could we make a composite target that incorporates the backend specific config.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/9) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/020892b0bb2509622e1cad07efa58dc7a794231fec3336480e355eb775cb84ee).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Junru Shao via Apache TVM Discuss <no...@discuss.tvm.ai>.
What about we define a new target kind:
```
{
"kind": "packaged", # probably need a better name, please propose new ones
"runtime": "crt", # the "runtime" in the proposal
"executor": { # the codegen target for relay function
# i.e. the "executor" in the proposal
"kind": "vm/aot",
...
},
"target": {
"kind": "cuda", # the target that TIR generates to
"host": {
"kind": "llvm", # the codegen target for the host-side driver code
...
}
},
}
```
We can provide helpers to sugar the construction of this recursive target:
```python
def tvm.target.packaged(
target="cuda",
executor="aot",
runtime="crt",
): ...
```
In the common case, user only need to feed with "cuda", because we could provide a good default. For advanced use cases, users could use the packaged API to specify their own specification for the package
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/36) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/b6de79fb08c829db7416cce66b8d996188ea6438d7307de95750016025848fa8).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Andrew Reusch via Apache TVM Discuss <no...@discuss.tvm.ai>.
Wow lots more discussion here! Thanks @junrushao1994 for writing up our discussions. So one thing I'd like to point out is that the recursive Target approach is not more expressive than the approach proposed by this original RFC. Expressing a "contains" relation can be done equivalently well by
- defining a recursion relationship inside the Target data structure
- defining another structure which describes the contains relationship (akin to a join table in database theory)
The main reason I am interested in the join-table approach here is that it vastly simplifies `MergeTarget` as described by Junru above. And, I'd like to point out that it's not sufficient here to merely define a function which hides the complexity under the covers. Users need to be able to understand what this function is doing because they are writing the inputs (though we are providing a tag, [Command Line Configuration Files](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0030-tvmc-comand-line-configuration-files.md) contemplates an expansion of the role of tagging to include tagging a partial configuration, as discussed earlier. I'm not sure it will be generally simple to explain how MergeTarget works as Target grows if we adopt the general approach of trying to attach every piece of compiler config to some Target which "owns" it.
The drawback of the flat configuration structure is it could be more difficult to consume inside the compiler. We should discuss whether this is truly an issue and how to mitigate it.
Finally, while I do think it's important to arrive at an an expressive, understandable Target data structure, as the compiler grows more complex, I think there is a tension between a Target structure which is clear to the user and a Target structure which naturally reflects the organization of the compiler (and therefore has the nice properties of clearly delineating where config should live and being easy to route in the compiler top-level). Hopefully, the organization of the compiler is also such that it's logical to a power user interested in creating a complex config. However, here I think that UX sugar can help to compose the common target patterns such as "cuda" (which really means 1 CUDA device with an implied "llvm" host). We already do this today anyway, so I suspect it will continue to play a role in the future.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/34) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/30809ee43fa1aa60868eecc5edabeb2520299736463ead505d5c1822243e6986).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Mark Shields via Apache TVM Discuss <no...@discuss.tvm.ai>.
Thumbs up from me, I'd like to see this proceed to PRs.
- Agree with adding config as a first-class field to IRModule.
- The build API backwards compat should be straightforward by extension of the existing `if isinstance` checks. We can emit deprecation warnings for a release or two.
- I think you want your tvmc cmdline flags to be interpreted as rewrites on CompilerConfig in some uniform way, right?
- We'll need to figure out some verbage for what should go into the config top level vs target etc. It's a significant risk the config turns into the union of misc fields across targets, passes etc. Many compilers end up in this world by accident and it's hard to unwind. But some principles and discipline up front should work.
- Not sure about how to reconcile this with the implicit contexts for Target and the PassConfigManager. Personally I'm not a fan of implicit context stacks but it is an established TVM patterns.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/3) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/5313b74333aa36277a8518b78d629f2a311babdaa0e187568d1d0166628b6277).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by tqchen via Apache TVM Discuss <no...@discuss.tvm.ai>.
cc @junrushao1994 @zxybazh @comaniac
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/2) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/db4f42cbf6d4a0aa8b9bbc82c7cd1dead0a9b5289c2ae354c4124a4b20aeeb29).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Christopher Sidebottom via Apache TVM Discuss <no...@discuss.tvm.ai>.
[quote="tqchen, post:29, topic:11372"]
This would results in two UX concepts. A target tag and config tag, and in the case of system implementations, possible two similar impls.
[/quote]
Which leads me to believe we should default to a `Config` level tag which is the highest level available?
[quote="tqchen, post:29, topic:11372"]
* Does the top-level config have to remain as in its most general form, e.g. can it be a `Union[Target,MultiTarget]`, as the most common case remains to be TargetWithHost
[/quote]
It would remain in the `Config` form on the `IRModule`, which means you could have either easily?
[quote="tqchen, post:29, topic:11372"]
* We might have need to propagate some of the multi-target constraint info in the future to function level, at that time point, which data structure to use(if there is both config and target).
[/quote]
Whichever is appropriate for the use-case, having standardised access to that information means you could access whichever is most useful to you. If you want to query the configuration for an appropriate `Target` and tag a function with it, that's an implementation detail of another part of the compiler.
[quote="tqchen, post:29, topic:11372"]
Right, this would result in two concepts, target and config. Both of which are really similar to each other and both can appear in the same automation logs. We might need to build two set of mechanisms for both if they end up as drastically different data structure without a common base.
[/quote]
Serialising of objects which don't share a common base is pretty common in many projects, and it's clear that `Configuration` encapsulates `Target` so can call the serialise internally? There's no need to complicate this by making everything a sub-class of `Target`. And I believe what @areusch was saying is that we didn't want anything but `Target` in the logs as it has no effect? Therefore encapsulating that with some function for creating logs from many pieces of the configuration may be useful?
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/31) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/7e03d8b9c7f992f89556bc686739231b6dcf11e69f05a628ef32b6e7c0fed85b).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by tqchen via Apache TVM Discuss <no...@discuss.tvm.ai>.
Independent from the engineering discussion. It would be useful to come back to the terminology and think about the UX consequence present that to the user. Obviously this is subjective, but worth to think about what can serve as a good story. I tried to search the term "Target" and "compiler target" on internet and here are some common interpretations:
- Target platform that the output runs on.
- Target program that we outputs.
"target platform" roughly aligns the first impression that comes up in my mind. A "platform" includes runtime libraries, hardwares, or any env that might affect the transformation behavior. Looking from that angle, "aws/c5" is certainly also a "target platform". A SoC with multiple chipsets, NPU, CPU can also certainly be viewed as a "target platform". A set of distributed machines can also be viewed as "target platform".
So then it goes back to what stories we tell our users(affects developers as well), and whether or not that story aligns with the most common sense impressions.
First kind of story:
- S0a: A target is specifies the "target platform", any deployment environment constraints that you are interested.
- S0b: If you are interested in multiple device settings, you can use a MultiTarget kind that composes up targets together and specifies the deployment env(that involves multiple devices).
Second kind of story:
- S1: A target is specifies the single device deployment constraints, you will need to compose them up to form a config, that also specifies the runtime and executor of your model.
S0 ties back to the common sense stories with a progression(first start with only target on single device, simple and easily receptive, then generalize by reusing the same concept that aligns). S1 would require more understanding in differentiating the concept, and resolving confusion that why a SoC with multiple chipset is not a "target platform".
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/30) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/b69ff9064589ff473671b9dfe45dce8e4b4c60440a1af9ac8c9f65f95c346e2c).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by tqchen via Apache TVM Discuss <no...@discuss.tvm.ai>.
Thanks @Mousius I am not suggesting a decision on solutions, but just want to broadly discuss the implication, of the engineering solutions. For example, to build on what you said
> Which leads me to believe we should default to a `Config` level tag which is the highest level available? If we add in-between layers then this still holds? For the same reason I wouldn’t want to use `Target` to define everything, I can see the error in trying to explain a `Target` tag and a `Config` tag and leading to confusion as to what a tag is - which I think we’re agreeing on
This would results in two UX concepts. A target tag and config tag, and in the case of system implementations, possible two similar impls.
> This RFC doesn’t aim to address how you use the configuration so much as define the fact the configuration will be there for you to use and rely on
I understand the original intention was to scope it as "top-level config". However, because config itself is a data structure(just like target) that involves "constraint settings throughout compilation", we naturally would to ask the following questions:
- Does the top-level config have to remain as in its most general form, e.g. can it be a `Union[Target,MultiTarget]`, as the most common case remains to be TargetWithHost
- We might have need to propagate some of the multi-target constraint info in the future to function level, at that time point, which data structure to use(if there is both config and target).
- The consistency of single device function's target attr and multi-device function's config attr.
> Are you suggesting something as simple as `configuration.to_json()` or `configuration.serialize_targets()` which would return the array of JSON represented `Target` ? Re-using the already defined schema for `Target` and providing some way to extract it seems to function here?
Right, this would result in two concepts, target and config. Both of which are really similar to each other and both can appear in the same automation logs. We might need to build two set of mechanisms for both if they end up as drastically different data structure without a common base.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/29) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/2ef6b123c86621f3fc20ad5de5704f36ea46ace576f121f46fadd8a94c532ed8).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Christopher Sidebottom via Apache TVM Discuss <no...@discuss.tvm.ai>.
[quote="tqchen, post:27, topic:11372"]
From N0’s pov, the ability to directly pass in Target with a host field is a good default solutions for this most comon combo, so in the case of API/UX design, we might want to encourage this kind of usage without worrying about additional fields for hetergenous setups in a config.
[/quote]
This was covered in the original post:
[quote="Mousius, post:1, topic:11372"]
From Python the module will be passed into C++ with the `CompilationConfig` to start the compilation flow:
```
def relay.build(ir_mod, config: Union[CompilationConfig, Target], ...):
if isinstance(config, Target):
config = CompilationConfig.from_target(config)
mod["build"](ir_mod, config)
```
[/quote]
[quote="tqchen, post:27, topic:11372"]
Under the context of config and target, we will need to be able to say that a tag can refers to either a config and Target, which effectively complicates the tagging system and explaination here. Additionally, there will be needs to have a common mechanism to register the tags for both target and config. Making them more uniformed would make this perspective more streamlined.
[/quote]
Ah, I understand, if we don't pass a `Target` and instead just pass a tag then you have to figure out which one to go for. The approach taken in [Command Line Configuration Files](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0030-tvmc-comand-line-configuration-files.md) is to wrap the `Target` in the JSON configuration. Which leads me to believe we should default to a `Config` level tag which is the highest level available? If we add in-between layers then this still holds? For the same reason I wouldn't want to use `Target` to define everything, I can see the error in trying to explain a `Target` tag and a `Config` tag and leading to confusion as to what a tag is - which I think we're agreeing on?
[quote="tqchen, post:27, topic:11372"]
From the N4’s pov, we will need to be able to represent the objects during decompositions, which means there will be need of smooth transitions of related information at the function level. For example, for some a function that involves mixed target host/device mixing the transitions to a device only. If that entails a difference in terms of the “target constraints”, e.g. for functions with multi-target it starts with a “config” attr, then for functions with a single device it becomes a “target” attr. Such transition is not as uniform.
[/quote]
This RFC doesn't aim to address how you use the configuration so much as define the fact the configuration will be there for you to use and rely on. [Unified device/target/memory scope planning](https://github.com/apache/tvm-rfcs/pull/38) stands out to me as an RFC which discusses how to correctly annotate a function for a specific use-case and other than providing a consistent view of the world the `CompilationConfig` does not impact this.
[quote="tqchen, post:27, topic:11372"]
In the context of N5, there will be a need to be able to log both single device target, or multitarget config as part of the autotuning logs in the same way. From the automation’s pov they are all “target constraints” of a function, or a collection of functions. As in N4, this would favor a single entity that captures the “target constraint” in an uniformed way, or at least a unified serialization mechanism and perhaps repr printing that covers the target involved.
[/quote]
Are you suggesting something as simple as `configuration.to_json()` or `configuration.serialize_targets()` which would return the array of JSON represented `Target` ? Re-using the already defined schema for `Target` and providing some way to extract it seems to function here?
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/28) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/d25387a57cd2b4ae3ef7325c5d6028eb188d7f8f26274c5207e90857725215c5).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by tqchen via Apache TVM Discuss <no...@discuss.tvm.ai>.
All the the alternatives (A1a, A1b, A1c), should be able to cover the need that we initially bought up -- around N3. Additionally, the Target system as it is now is already powerful enough to resolve the N3 related needs that was bought up, as the alternatives @junrushao1994 listed along the A1c direction.
In all cases , it is certainly possible to resolve the problems with extra layers of abstractions and indirections. As a matter of fact, they are all very similar, except for how the data structure itself is built up.
So the main thing that would be helpful here is to understand the tradeoffs here under different contexts, given our previous discussions was focused around N3, it is also helpful to look at things from other needs.
To give some examples:
From N0's pov, the ability to directly pass in Target with a host field is a good default solutions for this most comon combo, so in the case of API/UX design, we might want to encourage this kind of usage without worrying about additional fields for hetergenous setups in a config.
```python
build(mod, Target("cuda", host="llvm"))
```
Additionally, the transition between E0 to E1 encourages a transition from Target with host field(that indicates a mixed host program) to a device only(without host).
From N2's perspective. `aws/c5` favors deployment target as a holistic thing(aka at the config level).
```
build(mod, "aws/c5")
```
Under the context of config and target, we will need to be able to say that a tag can refers to either a config and Target, which effectively complicates the tagging system and explaination here. Additionally, there will be needs to have a common mechanism to register the tags for both target and config. Making them more uniformed would make this perspective more streamlined.
From the N4's pov, we will need to be able to represent the objects during decompositions, which means there will be need of smooth transitions of related information at the function level. For example, for some a function that involves mixed target host/device mixing the transitions to a device only. If that entails a difference in terms of the "target constraints", e.g. for functions with multi-target it starts with a "config" attr, then for functions with a single device it becomes a "target" attr. Such transition is not as uniform.
In the context of N5, there will be a need to be able to log both single device target, or multitarget config as part of the autotuning logs in the same way. From the automation's pov they are all "target constraints" of a function, or a collection of functions. As in N4, this would favor a single entity that captures the "target constraint" in an uniformed way, or at least a unified serialization mechanism and perhaps repr printing that covers the target involved.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/27) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/f0bd693e141f56bca8a52bb7db681478f6e6181f203a28a925f4c093188f6af5).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Christopher Sidebottom via Apache TVM Discuss <no...@discuss.tvm.ai>.
Hi @tqchen,
Reading through the various needs there's nothing which hasn't already been covered by this RFC in combination with already accepted RFCs. Could you articulate the next steps?
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/26) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/dd5e887bdb95fb1944a27ddbacf8204a01a2515dbe462c08ebcdb8187e85a84d).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Christopher Sidebottom via Apache TVM Discuss <no...@discuss.tvm.ai>.
[quote="junrushao1994, post:32, topic:11372"]
On the other hand, my concern is the fragmentation of APIs. It has been a huge problem in the recent 1-2 years, and we do have alternatives not to do so.
[/quote]
Could you elaborate on this? I believe this isn't solely a UX issue but also a hygiene factor within the compiler and how we represent the data structures internally so would rather not overload `Target` with `Executor` and `Runtime`. This RFC is proposing a suitable home for information that's relevant across the compilation given we now have at least `Executor` and `Runtime` to include, but a side effect is bringing the `tvmc` view back into alignment with the internals of the compiler.
It's also worth noting, that with the current RFC for [Migrating Target Attributes to IRModule](https://github.com/apache/tvm-rfcs/blob/main/rfcs/0029-migrating-to-irmodule-attributes.md) `tvmc` can glue this together with the relevant pieces, so from a user point of view they wouldn't know how disparate the internals are but it would be a headache to maintain.
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/33) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/c023e8fa7d47fe75c861095fe5ea6a5ab2af2a9ffb46265c5b74f106be07c850).
[Apache TVM Discuss] [Development/pre-RFC] [pre-RFC] Compilation
Configuration Representation
Posted by Junru Shao via Apache TVM Discuss <no...@discuss.tvm.ai>.
@areusch and I had long discussion yesterday offline, and he helped me understand the concern from the UX perspective: If we fold executor into target, then it's more difficult to separate the config coming from two parties, where one party impl the codegen and the other impl the executor.
On the other hand, my concern is the fragmentation of APIs. It has been a huge problem in the recent 1-2 years, and we do have alternatives not to do so.
Here is my proposal:
- Part 1. Add Exector/Runtime fields to TargetNode:
```
class TargetNode {
...
Executor executor;
Runtime runtime;
};
class Executor {
FromJSON();
AsJSON();
};
class Runtime {
FromJSON();
AsJSON();
};
```
- Part 2. Add a helper API to merge Target, Executor and Runtime
```
Target MergeTarget(Target target_without_executor_runtime, Executor executor, Runtime runtime);
```
- Part 3. Allow separate specification of target, target_host, executor, runtime in TVMC, and internally use the proposed API in Part 2 to merge, validate and normalize them into a single Target object
```
tvmc --target "llvm" --executor "..." --runtime "..."
```
- Part 4. For heterogeneous case, annotate the target onto each PrimFunc/RelayFunc to specify the target/runtime/executor
```
@tvm.script.ir_module
class Module:
@T.func
def tir_func():
T.func_attrs({"target": JSON-Repr-of-Target-Obj}) # with runtime&executor included
...
@R.func
def relay_func():
T.func_attrs({"target": JSON-Repr-of-Target-Obj}) # with runtime&executor included
...
```
---
[Visit Topic](https://discuss.tvm.apache.org/t/pre-rfc-compilation-configuration-representation/11372/32) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/69ecb26051e749ac91f3e6e2299eb2cb080b9e9e2996b8d2ea4ede093cdeb737).