You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/12/13 03:13:21 UTC
[GitHub] [tvm] masahi opened a new pull request #9721: [Relay, BYOC] Make constant binding in PartitionGraph optional
masahi opened a new pull request #9721:
URL: https://github.com/apache/tvm/pull/9721
As discussed in https://discuss.tvm.apache.org/t/byoc-cutlass-dealing-with-constants-in-c-source-gen-based-byoc/11362, constant binding in C-source based BYOC is a common problem. So far, I've been avoiding this problem by **not** running `bind_params` and `FoldConstant` before partitioning. However, to convert `conv2d -> batch_norm -> relu` into `conv2d -> bias_add -> relu`, constant folding is necessary.
This PR is my proposal to allow batch norm fusion while avoiding constants binding. The change is simple and works perfectly for my use case.
Before `PartitionGraph`:
```
%26 = annotation.compiler_begin(%input0, compiler="cutlass") /* ty=Tensor[(8, 3, 224, 224), float32] */;
%27 = annotation.compiler_begin(meta[relay.Constant][0] /* ty=Tensor[(64, 3, 7, 7), float32] */, compiler="cutlass") /* ty=Tensor[(64, 3, 7, 7), float32] */;
%28 = annotation.compiler_begin(meta[relay.Constant][1] /* ty=Tensor[(64, 1, 1), float32] */, compiler="cutlass") /* ty=Tensor[(64, 1, 1), float32] */;
%29 = fn (%FunctionVar_8_01: Tensor[(8, 3, 224, 224), float32], %FunctionVar_8_11: Tensor[(64, 3, 7, 7), float32], %FunctionVar_8_21: Tensor[(64, 1, 1), float32], PartitionedFromPattern="nn.conv2d_add_nn.relu_", Composite="cutlass.conv2d_bias_relu") -> Tensor[(8, 64, 112, 112), float32] {
...
};
%30 = %29(%26, %27, %28) /* ty=Tensor[(8, 64, 112, 112), float32] */;
```
After `PartitionGraph` with `bind_constants = False`:
```
def @main(%input0: Tensor[(8, 3, 224, 224), float32]) -> Tensor[(8, 1000), float32] {
%0 = @tvmgen_default_cutlass_main_0(%input0, meta[relay.Constant][0] /* ty=Tensor[(64, 3, 7, 7), float32] */, meta[relay.Constant][1] /* ty=Tensor[(64, 1, 1), float32] */) /* ty=Tensor[(8, 64, 112, 112), float32] */;
...
}
def @tvmgen_default_cutlass_main_0(%cutlass_0_i0: Tensor[(8, 3, 224, 224), float32], %cutlass_0_i1: Tensor[(64, 3, 7, 7), float32], %cutlass_0_i2: Tensor[(64, 1, 1), float32], Inline=1, Compiler="cutlass", global_symbol="tvmgen_default_cutlass_main_0", Primitive=1) -> Tensor[(8, 64, 112, 112), float32] {
%43 = fn (%FunctionVar_8_0: Tensor[(8, 3, 224, 224), float32], %FunctionVar_8_1: Tensor[(64, 3, 7, 7), float32], %FunctionVar_8_2: Tensor[(64, 1, 1), float32], PartitionedFromPattern="nn.conv2d_add_nn.relu_", Composite="cutlass.conv2d_bias_relu") -> Tensor[(8, 64, 112, 112), float32] {
...
};
%43(%cutlass_0_i0, %cutlass_0_i1, %cutlass_0_i2) /* ty=Tensor[(8, 64, 112, 112), float32] */
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #9721: [Relay, BYOC] Make constant binding in PartitionGraph optional
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #9721:
URL: https://github.com/apache/tvm/pull/9721#discussion_r768109935
##########
File path: python/tvm/relay/transform/transform.py
##########
@@ -695,17 +695,23 @@ def LambdaLift():
return _ffi_api.LambdaLift()
-def PartitionGraph(mod_name="default"):
+def PartitionGraph(mod_name="default", bind_constants=True):
"""Partition a Relay program into regions that can be executed on different
backends.
+ Parameters
+ ----------
+ bind_constants: bool
+ Whether or not to bind constants in partitioned subgraphs. For C-source based codegen,
+ it is recommended to set this to False to avoid embedding large constants in
+ a C source file.
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #9721: [Relay, BYOC] Make constant binding in PartitionGraph optional
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #9721:
URL: https://github.com/apache/tvm/pull/9721#discussion_r767983007
##########
File path: python/tvm/relay/transform/transform.py
##########
@@ -695,17 +695,23 @@ def LambdaLift():
return _ffi_api.LambdaLift()
-def PartitionGraph(mod_name="default"):
+def PartitionGraph(mod_name="default", bind_constants=True):
"""Partition a Relay program into regions that can be executed on different
backends.
+ Parameters
+ ----------
+ bind_constants: bool
+ Whether or not to bind constants in partitioned subgraphs. For C-source based codegen,
+ it is recommended to set this to False to avoid embedding large constants in
+ a C source file.
Review comment:
I made some changes to the description but please feel free to take any piece you prefer. Also could you also help adding the docstring for `mod_name`? It was missed somehow...
```
Parameters
----------
bind_constants: bool
Whether or not to bind constants in partitioned subgraphs. Note that the codegen needs
to maintain the bind constants; otherwise the constants will be maintained by
the metadata module. So it is recommended for C-source based codegens to
set bind_constants=False to avoid embedding large constants in a C source file.
```
##########
File path: src/relay/transforms/partition_graph.cc
##########
@@ -601,9 +607,10 @@ Pass PartitionGraph(String mod_name) {
{flatten_tuples_pass, remove_default_pass, partition_pass, name_mangling_pass, InferType()});
}
-TVM_REGISTER_GLOBAL("relay._transform.PartitionGraph").set_body_typed([](String mod_name) {
- return transform::PartitionGraph(mod_name);
-});
+TVM_REGISTER_GLOBAL("relay._transform.PartitionGraph")
+ .set_body_typed([](String mod_name, bool bind_constants = false) {
Review comment:
Just double check, does `set_body_typed` support default argument value? I sort of remember that we have to use `.set_body` and manually check the size of `TVMArgs` to deal with default values, but I might be wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #9721: [Relay, BYOC] Make constant binding in PartitionGraph optional
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #9721:
URL: https://github.com/apache/tvm/pull/9721#discussion_r768112083
##########
File path: src/relay/transforms/partition_graph.cc
##########
@@ -601,9 +607,10 @@ Pass PartitionGraph(String mod_name) {
{flatten_tuples_pass, remove_default_pass, partition_pass, name_mangling_pass, InferType()});
}
-TVM_REGISTER_GLOBAL("relay._transform.PartitionGraph").set_body_typed([](String mod_name) {
- return transform::PartitionGraph(mod_name);
-});
+TVM_REGISTER_GLOBAL("relay._transform.PartitionGraph")
+ .set_body_typed([](String mod_name, bool bind_constants = false) {
Review comment:
You are right, it seems removing `bind_constants` from `return _ffi_api.PartitionGraph(mod_name, bind_constants)` results in the error:
``` File "/home/masa/projects/dev/tvm/python/tvm/relay/transform/transform.py", line 721, in PartitionGraph
return _ffi_api.PartitionGraph(mod_name)
File "/home/masa/projects/dev/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
1: TVMFuncCall
0: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TypedPackedFunc<tvm::transform::Pass (tvm::runtime::String, bool)>::AssignTypedLambda<tvm::relay::transform::{lambda(tvm::runtime::String, bool)#1}>(tvm::relay::transform::{lambda(tvm::runtime::String, bool)#1}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
File "/home/masa/projects/dev/tvm/include/tvm/runtime/packed_func.h", line 1477
TVMError: Function relay._transform.PartitionGraph expects 2 arguments, but 1 were provided.
```
Removed the default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a change in pull request #9721: [Relay, BYOC] Make constant binding in PartitionGraph optional
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #9721:
URL: https://github.com/apache/tvm/pull/9721#discussion_r768112083
##########
File path: src/relay/transforms/partition_graph.cc
##########
@@ -601,9 +607,10 @@ Pass PartitionGraph(String mod_name) {
{flatten_tuples_pass, remove_default_pass, partition_pass, name_mangling_pass, InferType()});
}
-TVM_REGISTER_GLOBAL("relay._transform.PartitionGraph").set_body_typed([](String mod_name) {
- return transform::PartitionGraph(mod_name);
-});
+TVM_REGISTER_GLOBAL("relay._transform.PartitionGraph")
+ .set_body_typed([](String mod_name, bool bind_constants = false) {
Review comment:
You are right, it seems removing `bind_constants` from `return _ffi_api.PartitionGraph(mod_name, bind_constants)` results in the error:
``` File "/home/masa/projects/dev/tvm/python/tvm/relay/transform/transform.py", line 721, in PartitionGraph
return _ffi_api.PartitionGraph(mod_name)
File "/home/masa/projects/dev/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
1: TVMFuncCall
0: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TypedPackedFunc<tvm::transform::Pass (tvm::runtime::String, bool)>::AssignTypedLambda<tvm::relay::transform::{lambda(tvm::runtime::String, bool)#1}>(tvm::relay::transform::{lambda(tvm::runtime::String, bool)#1}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
File "/home/masa/projects/dev/tvm/include/tvm/runtime/packed_func.h", line 1477
TVMError: Function relay._transform.PartitionGraph expects 2 arguments, but 1 were provided.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi merged pull request #9721: [Relay, BYOC] Make constant binding in PartitionGraph optional
Posted by GitBox <gi...@apache.org>.
masahi merged pull request #9721:
URL: https://github.com/apache/tvm/pull/9721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org