You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/01/19 01:24:58 UTC
[GitHub] [incubator-tvm] masahi opened a new pull request #4741:
[Partitioning] Add test cases for fused ops with manual annotation
masahi opened a new pull request #4741: [Partitioning] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741
* A custom annotator which detects conv + bias add + relu ops
* An example of applying FoldScaleAxis and FoldConstant to layers of conv + bn + relu to get conv + bias add + relu ops which the annotator can detect (before partitioning)
* Test cases on a simple network and mobilenet which verify that partitioning is working as expected.
The result of partitioning mobilenet is dumped [here](https://gist.github.com/masahi/df0619d860b61b3cf9d7faad36e43fd7).
The actual execution of partitioned graphs is disabled due to the runtime issue described [in the forum](https://discuss.tvm.ai/t/external-codegen-how-the-runtime-determines-function-signatures-for-generated-functions/5455/).
please review @zhiics @comaniac
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi closed pull request #4741: [External
codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi closed pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on a change in pull request #4741:
[External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368707692
##########
File path: src/relay/backend/contrib/dnnl/codegen.cc
##########
@@ -50,82 +51,109 @@ class CodegenDNNL : public ExprVisitor, public CodegenCBase {
out_.push_back({node->name_hint(), 0});
}
- void VisitExpr_(const TupleGetItemNode* op) final {
- // Do nothing
- }
-
void VisitExpr_(const CallNode* call) final {
- std::ostringstream decl_stream;
- std::ostringstream buf_stream;
- // Args: ID
- std::vector<std::string> args;
+ struct Output {
+ std::string decl, buf;
+ int out_size = 1;
+ std::string out;
+ };
+
+ auto generate_body = [=](const CallNode* root_call, const std::string& func_name,
+ const std::vector<std::string>& args,
+ const std::vector<std::string>& fused_func_args) {
+ // Make function call with input buffers when visiting arguments
+ bool first = true;
+ std::ostringstream arg_stream;
+ arg_stream << "(";
+ for (size_t i = 0; i < root_call->args.size(); ++i) {
+ VisitExpr(root_call->args[i]);
+ for (auto out : out_) {
+ if (!first) {
+ arg_stream << ", ";
+ }
+ first = false;
+ arg_stream << out.first;
+ }
+ }
+
+ for (auto arg_name : fused_func_args) {
+ arg_stream << ", " << arg_name;
+ }
+
+ // Analyze the output buffer
+ auto type_node = root_call->checked_type().as<TensorTypeNode>();
+ CHECK(type_node != nullptr && runtime::TypeMatch(type_node->dtype, kDLFloat, 32))
+ << "Only support single output tensor with float type";
+
+ auto out_shape = GetShape(root_call->checked_type());
+
+ Output ret;
+ ret.out = "buf_" + std::to_string(buf_idx_++);
+ ret.out_size = std::accumulate(out_shape.begin(), out_shape.end(), 1, std::multiplies<int>());
+
+ this->PrintIndents();
+
+ std::ostringstream buf_stream;
+ buf_stream << "float* " << ret.out << " = (float*)std::malloc(4 * " << ret.out_size << ");";
+ ret.buf = buf_stream.str();
- // Get the arguments for various DNNL kernels.
- if (IsOp(call, "nn.conv2d")) {
- decl_stream << "dnnl_conv2d";
- args = Conv2d(call);
+ arg_stream << ", " << ret.out;
+ // Attach attribute arguments
+ for (size_t i = 0; i < args.size(); ++i) {
+ arg_stream << ", " << args[i];
+ }
+ arg_stream << ");";
+ ret.decl = func_name + arg_stream.str();
+
+ return ret;
+ };
+
+ Output ret;
+ if (auto conv_call = DetectFusedConv2DBiasReLU(call)) {
Review comment:
I can also leave the current dumb implementation as it is, with the understanding that
* This is a temporary solution
* It will serve as a concrete motivation and test case for validating a more general mechanism to be introduced
Trying to be a bit more clever and duplicating an entire state machine logic here do not seem worth it to me anymore. Either way I'm fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on a change in pull request #4741:
[External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368310856
##########
File path: src/relay/backend/contrib/dnnl/codegen.cc
##########
@@ -50,82 +51,109 @@ class CodegenDNNL : public ExprVisitor, public CodegenCBase {
out_.push_back({node->name_hint(), 0});
}
- void VisitExpr_(const TupleGetItemNode* op) final {
- // Do nothing
- }
-
void VisitExpr_(const CallNode* call) final {
- std::ostringstream decl_stream;
- std::ostringstream buf_stream;
- // Args: ID
- std::vector<std::string> args;
+ struct Output {
+ std::string decl, buf;
+ int out_size = 1;
+ std::string out;
+ };
+
+ auto generate_body = [=](const CallNode* root_call, const std::string& func_name,
+ const std::vector<std::string>& args,
+ const std::vector<std::string>& fused_func_args) {
+ // Make function call with input buffers when visiting arguments
+ bool first = true;
+ std::ostringstream arg_stream;
+ arg_stream << "(";
+ for (size_t i = 0; i < root_call->args.size(); ++i) {
+ VisitExpr(root_call->args[i]);
+ for (auto out : out_) {
+ if (!first) {
+ arg_stream << ", ";
+ }
+ first = false;
+ arg_stream << out.first;
+ }
+ }
+
+ for (auto arg_name : fused_func_args) {
+ arg_stream << ", " << arg_name;
+ }
+
+ // Analyze the output buffer
+ auto type_node = root_call->checked_type().as<TensorTypeNode>();
+ CHECK(type_node != nullptr && runtime::TypeMatch(type_node->dtype, kDLFloat, 32))
+ << "Only support single output tensor with float type";
+
+ auto out_shape = GetShape(root_call->checked_type());
+
+ Output ret;
+ ret.out = "buf_" + std::to_string(buf_idx_++);
+ ret.out_size = std::accumulate(out_shape.begin(), out_shape.end(), 1, std::multiplies<int>());
+
+ this->PrintIndents();
+
+ std::ostringstream buf_stream;
+ buf_stream << "float* " << ret.out << " = (float*)std::malloc(4 * " << ret.out_size << ");";
+ ret.buf = buf_stream.str();
- // Get the arguments for various DNNL kernels.
- if (IsOp(call, "nn.conv2d")) {
- decl_stream << "dnnl_conv2d";
- args = Conv2d(call);
+ arg_stream << ", " << ret.out;
+ // Attach attribute arguments
+ for (size_t i = 0; i < args.size(); ++i) {
+ arg_stream << ", " << args[i];
+ }
+ arg_stream << ");";
+ ret.decl = func_name + arg_stream.str();
+
+ return ret;
+ };
+
+ Output ret;
+ if (auto conv_call = DetectFusedConv2DBiasReLU(call)) {
Review comment:
The idea is for it to serve as an example of handling fused ops inside external codegen. I assume dnnl backend itself is not meant to be used in production; The purpose is to be a more realistic example than CodegenC, so I thought why don't we add an example of how to handle fused ops. I never intended to cover other fusion cases.
Since we are trying to be so nice to new backend implementers (who might not be familiar with TVM internals) as to add convenient op level annotation and semi automatic fusion mechanism etc for them, I don't think it is reasonable to expect them to figure out how to handle more complicated but often common cases (like fusion) and everything else on their own. Hope this make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4741:
[External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368693340
##########
File path: src/relay/backend/contrib/dnnl/codegen.cc
##########
@@ -50,82 +51,109 @@ class CodegenDNNL : public ExprVisitor, public CodegenCBase {
out_.push_back({node->name_hint(), 0});
}
- void VisitExpr_(const TupleGetItemNode* op) final {
- // Do nothing
- }
-
void VisitExpr_(const CallNode* call) final {
- std::ostringstream decl_stream;
- std::ostringstream buf_stream;
- // Args: ID
- std::vector<std::string> args;
+ struct Output {
+ std::string decl, buf;
+ int out_size = 1;
+ std::string out;
+ };
+
+ auto generate_body = [=](const CallNode* root_call, const std::string& func_name,
+ const std::vector<std::string>& args,
+ const std::vector<std::string>& fused_func_args) {
+ // Make function call with input buffers when visiting arguments
+ bool first = true;
+ std::ostringstream arg_stream;
+ arg_stream << "(";
+ for (size_t i = 0; i < root_call->args.size(); ++i) {
+ VisitExpr(root_call->args[i]);
+ for (auto out : out_) {
+ if (!first) {
+ arg_stream << ", ";
+ }
+ first = false;
+ arg_stream << out.first;
+ }
+ }
+
+ for (auto arg_name : fused_func_args) {
+ arg_stream << ", " << arg_name;
+ }
+
+ // Analyze the output buffer
+ auto type_node = root_call->checked_type().as<TensorTypeNode>();
+ CHECK(type_node != nullptr && runtime::TypeMatch(type_node->dtype, kDLFloat, 32))
+ << "Only support single output tensor with float type";
+
+ auto out_shape = GetShape(root_call->checked_type());
+
+ Output ret;
+ ret.out = "buf_" + std::to_string(buf_idx_++);
+ ret.out_size = std::accumulate(out_shape.begin(), out_shape.end(), 1, std::multiplies<int>());
+
+ this->PrintIndents();
+
+ std::ostringstream buf_stream;
+ buf_stream << "float* " << ret.out << " = (float*)std::malloc(4 * " << ret.out_size << ");";
+ ret.buf = buf_stream.str();
- // Get the arguments for various DNNL kernels.
- if (IsOp(call, "nn.conv2d")) {
- decl_stream << "dnnl_conv2d";
- args = Conv2d(call);
+ arg_stream << ", " << ret.out;
+ // Attach attribute arguments
+ for (size_t i = 0; i < args.size(); ++i) {
+ arg_stream << ", " << args[i];
+ }
+ arg_stream << ");";
+ ret.decl = func_name + arg_stream.str();
+
+ return ret;
+ };
+
+ Output ret;
+ if (auto conv_call = DetectFusedConv2DBiasReLU(call)) {
Review comment:
Yeah, I had a brief discussion with @u99127 before. I will read the discussion more carefully and probably we can discuss from there and try to have some consensus on a design/implementation. Sorry for being late/slow because I am on vacation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi edited a comment on issue #4741: [External
codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi edited a comment on issue #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#issuecomment-576030113
@zhiics I'm not trying to make DNNL backend more feature complete. I want to add examples and test cases of typical usage scenarios that most backend implementers are likely to encounter.
We talked on the forum that fusion is already possible with manual annotation. But there is no example which demonstrates that. This PR fill this gap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on issue #4741: [External codegen]
Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on issue #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#issuecomment-576030113
@zhiics I'm not trying to make DNNL backend more feature complete. I want to add examples and test cases of typical usage scenarios that most backend implementers are likely to encounter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on issue #4741: [External codegen]
Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on issue #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#issuecomment-576039103
I add a link below where I clarified my intention. Hopefully this clears up some confusion.
https://discuss.tvm.ai/t/solved-external-codegen-how-the-runtime-determines-function-signatures-for-generated-functions/5455/6
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] comaniac commented on issue #4741: [External
codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
comaniac commented on issue #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#issuecomment-584359173
As #4771 has been merged, we can revisit this PR for DNNL fuse patterns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on a change in pull request #4741:
[External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368503405
##########
File path: src/relay/backend/contrib/dnnl/codegen.cc
##########
@@ -50,82 +51,109 @@ class CodegenDNNL : public ExprVisitor, public CodegenCBase {
out_.push_back({node->name_hint(), 0});
}
- void VisitExpr_(const TupleGetItemNode* op) final {
- // Do nothing
- }
-
void VisitExpr_(const CallNode* call) final {
- std::ostringstream decl_stream;
- std::ostringstream buf_stream;
- // Args: ID
- std::vector<std::string> args;
+ struct Output {
+ std::string decl, buf;
+ int out_size = 1;
+ std::string out;
+ };
+
+ auto generate_body = [=](const CallNode* root_call, const std::string& func_name,
+ const std::vector<std::string>& args,
+ const std::vector<std::string>& fused_func_args) {
+ // Make function call with input buffers when visiting arguments
+ bool first = true;
+ std::ostringstream arg_stream;
+ arg_stream << "(";
+ for (size_t i = 0; i < root_call->args.size(); ++i) {
+ VisitExpr(root_call->args[i]);
+ for (auto out : out_) {
+ if (!first) {
+ arg_stream << ", ";
+ }
+ first = false;
+ arg_stream << out.first;
+ }
+ }
+
+ for (auto arg_name : fused_func_args) {
+ arg_stream << ", " << arg_name;
+ }
+
+ // Analyze the output buffer
+ auto type_node = root_call->checked_type().as<TensorTypeNode>();
+ CHECK(type_node != nullptr && runtime::TypeMatch(type_node->dtype, kDLFloat, 32))
+ << "Only support single output tensor with float type";
+
+ auto out_shape = GetShape(root_call->checked_type());
+
+ Output ret;
+ ret.out = "buf_" + std::to_string(buf_idx_++);
+ ret.out_size = std::accumulate(out_shape.begin(), out_shape.end(), 1, std::multiplies<int>());
+
+ this->PrintIndents();
+
+ std::ostringstream buf_stream;
+ buf_stream << "float* " << ret.out << " = (float*)std::malloc(4 * " << ret.out_size << ");";
+ ret.buf = buf_stream.str();
- // Get the arguments for various DNNL kernels.
- if (IsOp(call, "nn.conv2d")) {
- decl_stream << "dnnl_conv2d";
- args = Conv2d(call);
+ arg_stream << ", " << ret.out;
+ // Attach attribute arguments
+ for (size_t i = 0; i < args.size(); ++i) {
+ arg_stream << ", " << args[i];
+ }
+ arg_stream << ");";
+ ret.decl = func_name + arg_stream.str();
+
+ return ret;
+ };
+
+ Output ret;
+ if (auto conv_call = DetectFusedConv2DBiasReLU(call)) {
Review comment:
I can go ahead and implement this, but that would duplicate pattern matching logic that I already have in my python annotator. That sounds bad and it would become a perfect anti-example mentioned in the RFC below :)
I think I should close this one and wait for a better solution to be ready. I will wait for your input for now @comaniac @zhiics
https://discuss.tvm.ai/t/rfc-external-codegen-defining-composite-relay-operators/5470/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi edited a comment on issue #4741: [External
codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi edited a comment on issue #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#issuecomment-584368589
yes I want to update this PR but we don't have a way to hook `Composite` and `Compiler` attributes yet, so I couldn't "see" a composite conv + bias + relu in CodegenDNNL atm. Refer to the comments below.
https://github.com/apache/incubator-tvm/pull/4771#issuecomment-578066583
https://github.com/apache/incubator-tvm/pull/4771#discussion_r377029670
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on a change in pull request #4741:
[External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368310856
##########
File path: src/relay/backend/contrib/dnnl/codegen.cc
##########
@@ -50,82 +51,109 @@ class CodegenDNNL : public ExprVisitor, public CodegenCBase {
out_.push_back({node->name_hint(), 0});
}
- void VisitExpr_(const TupleGetItemNode* op) final {
- // Do nothing
- }
-
void VisitExpr_(const CallNode* call) final {
- std::ostringstream decl_stream;
- std::ostringstream buf_stream;
- // Args: ID
- std::vector<std::string> args;
+ struct Output {
+ std::string decl, buf;
+ int out_size = 1;
+ std::string out;
+ };
+
+ auto generate_body = [=](const CallNode* root_call, const std::string& func_name,
+ const std::vector<std::string>& args,
+ const std::vector<std::string>& fused_func_args) {
+ // Make function call with input buffers when visiting arguments
+ bool first = true;
+ std::ostringstream arg_stream;
+ arg_stream << "(";
+ for (size_t i = 0; i < root_call->args.size(); ++i) {
+ VisitExpr(root_call->args[i]);
+ for (auto out : out_) {
+ if (!first) {
+ arg_stream << ", ";
+ }
+ first = false;
+ arg_stream << out.first;
+ }
+ }
+
+ for (auto arg_name : fused_func_args) {
+ arg_stream << ", " << arg_name;
+ }
+
+ // Analyze the output buffer
+ auto type_node = root_call->checked_type().as<TensorTypeNode>();
+ CHECK(type_node != nullptr && runtime::TypeMatch(type_node->dtype, kDLFloat, 32))
+ << "Only support single output tensor with float type";
+
+ auto out_shape = GetShape(root_call->checked_type());
+
+ Output ret;
+ ret.out = "buf_" + std::to_string(buf_idx_++);
+ ret.out_size = std::accumulate(out_shape.begin(), out_shape.end(), 1, std::multiplies<int>());
+
+ this->PrintIndents();
+
+ std::ostringstream buf_stream;
+ buf_stream << "float* " << ret.out << " = (float*)std::malloc(4 * " << ret.out_size << ");";
+ ret.buf = buf_stream.str();
- // Get the arguments for various DNNL kernels.
- if (IsOp(call, "nn.conv2d")) {
- decl_stream << "dnnl_conv2d";
- args = Conv2d(call);
+ arg_stream << ", " << ret.out;
+ // Attach attribute arguments
+ for (size_t i = 0; i < args.size(); ++i) {
+ arg_stream << ", " << args[i];
+ }
+ arg_stream << ");";
+ ret.decl = func_name + arg_stream.str();
+
+ return ret;
+ };
+
+ Output ret;
+ if (auto conv_call = DetectFusedConv2DBiasReLU(call)) {
Review comment:
The idea is for it to serve as an example of handling fused ops inside external codegen. I assume dnnl backend itself is not meant to be used in production; The purpose is to be a more realistic example than CodegenC, so I thought why don't we add an example of how to handle fused ops. I never intended to cover other fusion cases.
Since we are trying to be so nice to new backend implementers (who might not be familiar with TVM internals) as to add convenient op level annotation and semi automatic fusion mechanism etc for them, I don't think it is reasonable to expect them to figure out how to handle more complicated but often common cases (like fusion) on their own. Hope this make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on issue #4741: [External codegen]
Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on issue #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#issuecomment-610731955
#5272
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] comaniac commented on a change in pull request
#4741: [External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368360176
##########
File path: tests/python/relay/test_pass_partition_graph.py
##########
@@ -425,10 +470,120 @@ def test_extern_dnnl_mobilenet():
(1, 1000), ref_res.asnumpy(), tol=1e-5, params=params)
+def test_partition_conv_bias_relu():
+ if not tvm.get_global_func("relay.ext.dnnl", True):
+ print("skip because DNNL codegen is not available")
+ return
+
+ def get_layers(prefix, data, in_channel, out_channel,
Review comment:
From its functionality, `get_block` might be more precise.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on a change in pull request #4741:
[External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368367461
##########
File path: src/relay/backend/contrib/dnnl/codegen.cc
##########
@@ -50,82 +51,109 @@ class CodegenDNNL : public ExprVisitor, public CodegenCBase {
out_.push_back({node->name_hint(), 0});
}
- void VisitExpr_(const TupleGetItemNode* op) final {
- // Do nothing
- }
-
void VisitExpr_(const CallNode* call) final {
- std::ostringstream decl_stream;
- std::ostringstream buf_stream;
- // Args: ID
- std::vector<std::string> args;
+ struct Output {
+ std::string decl, buf;
+ int out_size = 1;
+ std::string out;
+ };
+
+ auto generate_body = [=](const CallNode* root_call, const std::string& func_name,
+ const std::vector<std::string>& args,
+ const std::vector<std::string>& fused_func_args) {
+ // Make function call with input buffers when visiting arguments
+ bool first = true;
+ std::ostringstream arg_stream;
+ arg_stream << "(";
+ for (size_t i = 0; i < root_call->args.size(); ++i) {
+ VisitExpr(root_call->args[i]);
+ for (auto out : out_) {
+ if (!first) {
+ arg_stream << ", ";
+ }
+ first = false;
+ arg_stream << out.first;
+ }
+ }
+
+ for (auto arg_name : fused_func_args) {
+ arg_stream << ", " << arg_name;
+ }
+
+ // Analyze the output buffer
+ auto type_node = root_call->checked_type().as<TensorTypeNode>();
+ CHECK(type_node != nullptr && runtime::TypeMatch(type_node->dtype, kDLFloat, 32))
+ << "Only support single output tensor with float type";
+
+ auto out_shape = GetShape(root_call->checked_type());
+
+ Output ret;
+ ret.out = "buf_" + std::to_string(buf_idx_++);
+ ret.out_size = std::accumulate(out_shape.begin(), out_shape.end(), 1, std::multiplies<int>());
+
+ this->PrintIndents();
+
+ std::ostringstream buf_stream;
+ buf_stream << "float* " << ret.out << " = (float*)std::malloc(4 * " << ret.out_size << ");";
+ ret.buf = buf_stream.str();
- // Get the arguments for various DNNL kernels.
- if (IsOp(call, "nn.conv2d")) {
- decl_stream << "dnnl_conv2d";
- args = Conv2d(call);
+ arg_stream << ", " << ret.out;
+ // Attach attribute arguments
+ for (size_t i = 0; i < args.size(); ++i) {
+ arg_stream << ", " << args[i];
+ }
+ arg_stream << ");";
+ ret.decl = func_name + arg_stream.str();
+
+ return ret;
+ };
+
+ Output ret;
+ if (auto conv_call = DetectFusedConv2DBiasReLU(call)) {
Review comment:
yeah, this is my minimal effort way to detect only the pattern I care about. Will think about how to make it more general.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on a change in pull request #4741:
[External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368311255
##########
File path: src/relay/backend/contrib/dnnl/codegen.cc
##########
@@ -50,82 +51,109 @@ class CodegenDNNL : public ExprVisitor, public CodegenCBase {
out_.push_back({node->name_hint(), 0});
}
- void VisitExpr_(const TupleGetItemNode* op) final {
- // Do nothing
- }
-
void VisitExpr_(const CallNode* call) final {
- std::ostringstream decl_stream;
- std::ostringstream buf_stream;
- // Args: ID
- std::vector<std::string> args;
+ struct Output {
+ std::string decl, buf;
+ int out_size = 1;
+ std::string out;
+ };
+
+ auto generate_body = [=](const CallNode* root_call, const std::string& func_name,
+ const std::vector<std::string>& args,
+ const std::vector<std::string>& fused_func_args) {
+ // Make function call with input buffers when visiting arguments
+ bool first = true;
+ std::ostringstream arg_stream;
+ arg_stream << "(";
+ for (size_t i = 0; i < root_call->args.size(); ++i) {
+ VisitExpr(root_call->args[i]);
+ for (auto out : out_) {
+ if (!first) {
+ arg_stream << ", ";
+ }
+ first = false;
+ arg_stream << out.first;
+ }
+ }
+
+ for (auto arg_name : fused_func_args) {
+ arg_stream << ", " << arg_name;
+ }
+
+ // Analyze the output buffer
+ auto type_node = root_call->checked_type().as<TensorTypeNode>();
+ CHECK(type_node != nullptr && runtime::TypeMatch(type_node->dtype, kDLFloat, 32))
+ << "Only support single output tensor with float type";
+
+ auto out_shape = GetShape(root_call->checked_type());
+
+ Output ret;
+ ret.out = "buf_" + std::to_string(buf_idx_++);
+ ret.out_size = std::accumulate(out_shape.begin(), out_shape.end(), 1, std::multiplies<int>());
+
+ this->PrintIndents();
+
+ std::ostringstream buf_stream;
+ buf_stream << "float* " << ret.out << " = (float*)std::malloc(4 * " << ret.out_size << ");";
+ ret.buf = buf_stream.str();
- // Get the arguments for various DNNL kernels.
- if (IsOp(call, "nn.conv2d")) {
- decl_stream << "dnnl_conv2d";
- args = Conv2d(call);
+ arg_stream << ", " << ret.out;
+ // Attach attribute arguments
+ for (size_t i = 0; i < args.size(); ++i) {
+ arg_stream << ", " << args[i];
+ }
+ arg_stream << ");";
+ ret.decl = func_name + arg_stream.str();
+
+ return ret;
+ };
+
+ Output ret;
+ if (auto conv_call = DetectFusedConv2DBiasReLU(call)) {
Review comment:
Another usage scenario which I think is going to be common is translation from quantized Relay models. It would be great to add an example of translating QNN subgraphs to backend implementations for example. Since DNNL has quantization support and everyone can use it, it would serve as a good example and test case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on a change in pull request #4741:
[External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368503840
##########
File path: python/tvm/relay/build_module.py
##########
@@ -296,6 +299,28 @@ def optimize(mod, target=None, params=None):
return mod, params
+def bind_params_by_name(func, params):
Review comment:
ok. I need to think about how to test this function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] comaniac commented on a change in pull request
#4741: [External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368361331
##########
File path: tests/python/relay/test_pass_partition_graph.py
##########
@@ -425,10 +470,120 @@ def test_extern_dnnl_mobilenet():
(1, 1000), ref_res.asnumpy(), tol=1e-5, params=params)
+def test_partition_conv_bias_relu():
+ if not tvm.get_global_func("relay.ext.dnnl", True):
+ print("skip because DNNL codegen is not available")
+ return
+
+ def get_layers(prefix, data, in_channel, out_channel,
+ include_bn=True, include_sigmoid=False):
+ weight = relay.var(prefix + "weight")
+ bn_gamma = relay.var(prefix + "bn_gamma")
+ bn_beta = relay.var(prefix + "bn_beta")
+ bn_mmean = relay.var(prefix + "bn_mean")
+ bn_mvar = relay.var(prefix + "bn_var")
+
+ layer = relay.nn.conv2d(data=data, weight=weight, kernel_size=(3, 3),
+ channels=out_channel, padding=(1, 1))
+ if include_bn:
+ bn_output = relay.nn.batch_norm(layer, bn_gamma, bn_beta,
+ bn_mmean, bn_mvar)
+ layer = bn_output[0]
+ if include_sigmoid:
+ # dummy layer to prevent pattern detection
+ layer = relay.sigmoid(layer)
+ layer = relay.nn.relu(layer)
+ return layer
+
+ def get_net(include_bn=True, include_sigmoid=False):
+ data = relay.var("data", relay.TensorType((1, 3, 224, 224), "float32"))
+ layer1 = get_layers("layer1_", data, 3, 16, include_bn, include_sigmoid)
+ layer2 = get_layers("layer2_", layer1, 16, 16, include_bn, include_sigmoid)
+ last = layer2
+ return relay.Function(relay.analysis.free_vars(last), last)
+
+ def pre_optimize(mod, params):
+ remove_bn_pass = transform.Sequential([
+ relay.transform.InferType(),
+ relay.transform.SimplifyInference(),
+ relay.transform.FoldConstant(),
+ relay.transform.FoldScaleAxis(),
+ ])
+
+ if params != {}:
+ # This is required for constant folding
+ mod["main"] = bind_params_by_name(mod["main"], params)
+
+ with relay.build_config(opt_level=3, disabled_pass=["AlterOpLayout"]):
+ mod = remove_bn_pass(mod)
+
+ return mod
+
+ def get_partitoned_mod(mod):
+ mod["main"] = ConvBiasAddReLUAnnotator("dnnl").visit(mod["main"])
+ mod = transform.PartitionGraph()(mod)
+ return mod
+
+ def get_partitions(mod):
+ partitions = []
+
+ def visit_func(expr):
+ if isinstance(expr, _expr.Function) and expr != mod["main"]:
+ partitions.append(expr)
+ analysis.post_order_visit(mod["main"], visit_func)
+ return partitions
+
+ def test_detect_pattern(include_bn, include_sigmoid, num_expected_partition):
+ net = get_net(include_bn, include_sigmoid)
+ mod, params = tvm.relay.testing.create_workload(net)
+ mod = pre_optimize(mod, params)
+ mod = get_partitoned_mod(mod)
+ assert(len(get_partitions(mod)) == num_expected_partition)
+
+ def test_partition():
+ # conv + bn + relu -> detection succeed
+ test_detect_pattern(True, False, 2)
+ # conv + relu -> fail
+ test_detect_pattern(False, False, 0)
+ # conv + bn + sigmoid + relu -> fail
+ test_detect_pattern(True, True, 0)
+
+ def test_partition_mobilenet():
+ mod, params = relay.testing.mobilenet.get_workload()
+ mod = pre_optimize(mod, params)
+ mod = get_partitoned_mod(mod)
+ assert(len(get_partitions(mod)) == 27)
+
+ def test_exec(mod, params, ref_mod, ref_params, out_shape):
+ ishape = (1, 3, 224, 224)
+ i_data = np.random.randn(*ishape).astype(np.float32)
+ ref_ex = relay.create_executor("graph", mod=ref_mod, ctx=tvm.cpu(0))
+ ref_res = ref_ex.evaluate()(i_data, **ref_params)
Review comment:
It's better to add `compile_engine.get().clear()`. Otherwise it's possible that the module doesn't really run with DNNL in `check_result` because ops in `mod` and `ref_mod` are the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi edited a comment on issue #4741: [External
codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi edited a comment on issue #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#issuecomment-576039103
I add a link below where I clarified my intention. Hopefully this clears up some confusion.
https://discuss.tvm.ai/t/solved-external-codegen-how-the-runtime-determines-function-signatures-for-generated-functions/5455/7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] comaniac commented on a change in pull request
#4741: [External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368362837
##########
File path: src/relay/backend/contrib/dnnl/codegen.cc
##########
@@ -50,82 +51,109 @@ class CodegenDNNL : public ExprVisitor, public CodegenCBase {
out_.push_back({node->name_hint(), 0});
}
- void VisitExpr_(const TupleGetItemNode* op) final {
- // Do nothing
- }
-
void VisitExpr_(const CallNode* call) final {
- std::ostringstream decl_stream;
- std::ostringstream buf_stream;
- // Args: ID
- std::vector<std::string> args;
+ struct Output {
+ std::string decl, buf;
+ int out_size = 1;
+ std::string out;
+ };
+
+ auto generate_body = [=](const CallNode* root_call, const std::string& func_name,
+ const std::vector<std::string>& args,
+ const std::vector<std::string>& fused_func_args) {
+ // Make function call with input buffers when visiting arguments
+ bool first = true;
+ std::ostringstream arg_stream;
+ arg_stream << "(";
+ for (size_t i = 0; i < root_call->args.size(); ++i) {
+ VisitExpr(root_call->args[i]);
+ for (auto out : out_) {
+ if (!first) {
+ arg_stream << ", ";
+ }
+ first = false;
+ arg_stream << out.first;
+ }
+ }
+
+ for (auto arg_name : fused_func_args) {
+ arg_stream << ", " << arg_name;
+ }
+
+ // Analyze the output buffer
+ auto type_node = root_call->checked_type().as<TensorTypeNode>();
+ CHECK(type_node != nullptr && runtime::TypeMatch(type_node->dtype, kDLFloat, 32))
+ << "Only support single output tensor with float type";
+
+ auto out_shape = GetShape(root_call->checked_type());
+
+ Output ret;
+ ret.out = "buf_" + std::to_string(buf_idx_++);
+ ret.out_size = std::accumulate(out_shape.begin(), out_shape.end(), 1, std::multiplies<int>());
+
+ this->PrintIndents();
+
+ std::ostringstream buf_stream;
+ buf_stream << "float* " << ret.out << " = (float*)std::malloc(4 * " << ret.out_size << ");";
+ ret.buf = buf_stream.str();
- // Get the arguments for various DNNL kernels.
- if (IsOp(call, "nn.conv2d")) {
- decl_stream << "dnnl_conv2d";
- args = Conv2d(call);
+ arg_stream << ", " << ret.out;
+ // Attach attribute arguments
+ for (size_t i = 0; i < args.size(); ++i) {
+ arg_stream << ", " << args[i];
+ }
+ arg_stream << ");";
+ ret.decl = func_name + arg_stream.str();
+
+ return ret;
+ };
+
+ Output ret;
+ if (auto conv_call = DetectFusedConv2DBiasReLU(call)) {
Review comment:
While I agree with you that it's fine to handle fusion in this DNNL codegen, I also agree with @zhiics that the current implementation is a bit too ad-hoc even it's only used for demo purpose for now. As you have implemented, MKL DNN uses `set_post_ops` to attach ops to be fused. I think this part could be more general. For example:
```
if call == "relu":
visit(arg)
if this->curr_layer == "conv2d":
generate_post_ops(call)
else:
generate_a_layer(call)
```
In this way, the codegen is able to deal with all MKL DNN supported conv2d fusion (conv2d, conv2d+add, conv2d+add+relu). We could still put heuristic pattern annotations to the annotator and improve it gradually. I like the one you made for conv2d+bias+relu in this PR, for instance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4741:
[External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368308285
##########
File path: src/relay/backend/contrib/dnnl/codegen.cc
##########
@@ -50,82 +51,109 @@ class CodegenDNNL : public ExprVisitor, public CodegenCBase {
out_.push_back({node->name_hint(), 0});
}
- void VisitExpr_(const TupleGetItemNode* op) final {
- // Do nothing
- }
-
void VisitExpr_(const CallNode* call) final {
- std::ostringstream decl_stream;
- std::ostringstream buf_stream;
- // Args: ID
- std::vector<std::string> args;
+ struct Output {
+ std::string decl, buf;
+ int out_size = 1;
+ std::string out;
+ };
+
+ auto generate_body = [=](const CallNode* root_call, const std::string& func_name,
+ const std::vector<std::string>& args,
+ const std::vector<std::string>& fused_func_args) {
+ // Make function call with input buffers when visiting arguments
+ bool first = true;
+ std::ostringstream arg_stream;
+ arg_stream << "(";
+ for (size_t i = 0; i < root_call->args.size(); ++i) {
+ VisitExpr(root_call->args[i]);
+ for (auto out : out_) {
+ if (!first) {
+ arg_stream << ", ";
+ }
+ first = false;
+ arg_stream << out.first;
+ }
+ }
+
+ for (auto arg_name : fused_func_args) {
+ arg_stream << ", " << arg_name;
+ }
+
+ // Analyze the output buffer
+ auto type_node = root_call->checked_type().as<TensorTypeNode>();
+ CHECK(type_node != nullptr && runtime::TypeMatch(type_node->dtype, kDLFloat, 32))
+ << "Only support single output tensor with float type";
+
+ auto out_shape = GetShape(root_call->checked_type());
+
+ Output ret;
+ ret.out = "buf_" + std::to_string(buf_idx_++);
+ ret.out_size = std::accumulate(out_shape.begin(), out_shape.end(), 1, std::multiplies<int>());
+
+ this->PrintIndents();
+
+ std::ostringstream buf_stream;
+ buf_stream << "float* " << ret.out << " = (float*)std::malloc(4 * " << ret.out_size << ");";
+ ret.buf = buf_stream.str();
- // Get the arguments for various DNNL kernels.
- if (IsOp(call, "nn.conv2d")) {
- decl_stream << "dnnl_conv2d";
- args = Conv2d(call);
+ arg_stream << ", " << ret.out;
+ // Attach attribute arguments
+ for (size_t i = 0; i < args.size(); ++i) {
+ arg_stream << ", " << args[i];
+ }
+ arg_stream << ");";
+ ret.decl = func_name + arg_stream.str();
+
+ return ret;
+ };
+
+ Output ret;
+ if (auto conv_call = DetectFusedConv2DBiasReLU(call)) {
Review comment:
I am not sure if we really want to handle fused op from relay for external codegen. This looks quite ad-hoc to me. You may have countless combinations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on issue #4741: [External codegen]
Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on issue #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#issuecomment-584368589
yes I want to update this PR but we don't have a way to hook `Composite` and `Compiler` attributes yet, so I couldn't "see" a composite conv + bias + relu in CodegenDNNL atm. Refer to the comment below.
https://github.com/apache/incubator-tvm/pull/4771#issuecomment-578066583
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] comaniac commented on a change in pull request
#4741: [External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368360353
##########
File path: tests/python/relay/test_pass_partition_graph.py
##########
@@ -425,10 +470,120 @@ def test_extern_dnnl_mobilenet():
(1, 1000), ref_res.asnumpy(), tol=1e-5, params=params)
+def test_partition_conv_bias_relu():
+ if not tvm.get_global_func("relay.ext.dnnl", True):
+ print("skip because DNNL codegen is not available")
+ return
+
+ def get_layers(prefix, data, in_channel, out_channel,
+ include_bn=True, include_sigmoid=False):
+ weight = relay.var(prefix + "weight")
+ bn_gamma = relay.var(prefix + "bn_gamma")
+ bn_beta = relay.var(prefix + "bn_beta")
+ bn_mmean = relay.var(prefix + "bn_mean")
+ bn_mvar = relay.var(prefix + "bn_var")
+
+ layer = relay.nn.conv2d(data=data, weight=weight, kernel_size=(3, 3),
+ channels=out_channel, padding=(1, 1))
+ if include_bn:
+ bn_output = relay.nn.batch_norm(layer, bn_gamma, bn_beta,
+ bn_mmean, bn_mvar)
+ layer = bn_output[0]
+ if include_sigmoid:
+ # dummy layer to prevent pattern detection
+ layer = relay.sigmoid(layer)
+ layer = relay.nn.relu(layer)
+ return layer
+
+ def get_net(include_bn=True, include_sigmoid=False):
+ data = relay.var("data", relay.TensorType((1, 3, 224, 224), "float32"))
+ layer1 = get_layers("layer1_", data, 3, 16, include_bn, include_sigmoid)
+ layer2 = get_layers("layer2_", layer1, 16, 16, include_bn, include_sigmoid)
+ last = layer2
+ return relay.Function(relay.analysis.free_vars(last), last)
+
+ def pre_optimize(mod, params):
+ remove_bn_pass = transform.Sequential([
+ relay.transform.InferType(),
+ relay.transform.SimplifyInference(),
+ relay.transform.FoldConstant(),
+ relay.transform.FoldScaleAxis(),
+ ])
+
+ if params != {}:
Review comment:
`if params:`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi commented on a change in pull request #4741:
[External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368311255
##########
File path: src/relay/backend/contrib/dnnl/codegen.cc
##########
@@ -50,82 +51,109 @@ class CodegenDNNL : public ExprVisitor, public CodegenCBase {
out_.push_back({node->name_hint(), 0});
}
- void VisitExpr_(const TupleGetItemNode* op) final {
- // Do nothing
- }
-
void VisitExpr_(const CallNode* call) final {
- std::ostringstream decl_stream;
- std::ostringstream buf_stream;
- // Args: ID
- std::vector<std::string> args;
+ struct Output {
+ std::string decl, buf;
+ int out_size = 1;
+ std::string out;
+ };
+
+ auto generate_body = [=](const CallNode* root_call, const std::string& func_name,
+ const std::vector<std::string>& args,
+ const std::vector<std::string>& fused_func_args) {
+ // Make function call with input buffers when visiting arguments
+ bool first = true;
+ std::ostringstream arg_stream;
+ arg_stream << "(";
+ for (size_t i = 0; i < root_call->args.size(); ++i) {
+ VisitExpr(root_call->args[i]);
+ for (auto out : out_) {
+ if (!first) {
+ arg_stream << ", ";
+ }
+ first = false;
+ arg_stream << out.first;
+ }
+ }
+
+ for (auto arg_name : fused_func_args) {
+ arg_stream << ", " << arg_name;
+ }
+
+ // Analyze the output buffer
+ auto type_node = root_call->checked_type().as<TensorTypeNode>();
+ CHECK(type_node != nullptr && runtime::TypeMatch(type_node->dtype, kDLFloat, 32))
+ << "Only support single output tensor with float type";
+
+ auto out_shape = GetShape(root_call->checked_type());
+
+ Output ret;
+ ret.out = "buf_" + std::to_string(buf_idx_++);
+ ret.out_size = std::accumulate(out_shape.begin(), out_shape.end(), 1, std::multiplies<int>());
+
+ this->PrintIndents();
+
+ std::ostringstream buf_stream;
+ buf_stream << "float* " << ret.out << " = (float*)std::malloc(4 * " << ret.out_size << ");";
+ ret.buf = buf_stream.str();
- // Get the arguments for various DNNL kernels.
- if (IsOp(call, "nn.conv2d")) {
- decl_stream << "dnnl_conv2d";
- args = Conv2d(call);
+ arg_stream << ", " << ret.out;
+ // Attach attribute arguments
+ for (size_t i = 0; i < args.size(); ++i) {
+ arg_stream << ", " << args[i];
+ }
+ arg_stream << ");";
+ ret.decl = func_name + arg_stream.str();
+
+ return ret;
+ };
+
+ Output ret;
+ if (auto conv_call = DetectFusedConv2DBiasReLU(call)) {
Review comment:
Another usage scenario which I think is going to be common is translation from quantized Relay models. It would be great to add an example of translating QNN subgraphs to backend implementations, for example. Without it, it is not obvious how to go about it.
Since DNNL has quantization support and everyone can use it, it would serve as a good example and test case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] mbarrett97 commented on a change in pull request
#4741: [External codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
mbarrett97 commented on a change in pull request #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#discussion_r368501020
##########
File path: python/tvm/relay/build_module.py
##########
@@ -296,6 +299,28 @@ def optimize(mod, target=None, params=None):
return mod, params
+def bind_params_by_name(func, params):
Review comment:
Could we move this change to expose bind_params_by_name into its own PR? I could also make use of this more generally so I don't think it's specific to this test case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] masahi edited a comment on issue #4741: [External
codegen] Add test cases for fused ops with manual annotation
Posted by GitBox <gi...@apache.org>.
masahi edited a comment on issue #4741: [External codegen] Add test cases for fused ops with manual annotation
URL: https://github.com/apache/incubator-tvm/pull/4741#issuecomment-576030113
@zhiics I'm not trying to make DNNL backend more feature complete. I want to add examples and test cases of typical usage scenarios that most backend implementers are likely to encounter.
We talked on the forum that fusion is already possible with manual annotation. But there is no example which demonstrates that. This PR full this gap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services