You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/06/23 12:37:25 UTC
[GitHub] [tvm] echuraev opened a new pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
echuraev opened a new pull request #8313:
URL: https://github.com/apache/tvm/pull/8313
The Metal has some limitations on the number of input parameters. More
information can be found here:
https://developer.apple.com/documentation/metal/buffers/about_argument_buffers?language=objc
In this commit a new pass for splitting functions with big number of
arguments to smaller parts was added. In parameter `max_function_args`
we can specify the maximum number of kernel arguments for specific
target and then split kernel when the number of arguments exceeds the
value of `max_function_args`. Currently this pass works only for concat
layer.
Thanks for contributing to TVM! Please refer to guideline https://tvm.apache.org/docs/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @ them in the pull request thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-870050776
Not exactly, but I've dealt with a similar issue. My mitigation was to limit the maximum fusion depth, which breaks large parameter kernels into smaller ones. But that is not guaranteed to work and not predictable. I can imagine that having a pass like this that allows more fine-grained controls might be necessary in some cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] jwfromm commented on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
jwfromm commented on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-869829030
@mbrookhart can you take a look at this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] mbrookhart commented on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
mbrookhart commented on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-870658270
I have no problems merging this as it stands, but I do think I have a bigger question:
Should we put some sort of logic into fusion to automatically stop fusion if the argument list grows too large per this setting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] echuraev commented on a change in pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
echuraev commented on a change in pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#discussion_r657054872
##########
File path: src/relay/transforms/split_args.cc
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file split_args.cc
+ */
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/transform.h>
+
+#include "pattern_utils.h"
+
+namespace tvm {
+namespace relay {
+
+class ArgumentSplitter : public ExprRewriter {
+ public:
+ explicit ArgumentSplitter(int max_function_args)
+ : max_function_args_(max_function_args), concat_op_(Op::Get("concatenate")) {}
+
+ Expr Rewrite_(const CallNode* call, const Expr& post) final {
+ if (max_function_args_ < 0) return post;
+ if (call->op == concat_op_) {
+ auto op = call->args[0].as<TupleNode>();
+ const auto param = call->attrs.as<ConcatenateAttrs>();
+ const int limit = max_function_args_ - 1; // one buffer with output
Review comment:
I have a doubt about this `- 1` here. But I didn't find how can I get the number of outputs from `relay::CallNode`. Maybe someone could help me, how can I get the number of outputs for the call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi edited a comment on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
masahi edited a comment on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-870050776
Not exactly, but I've dealt with a similar issue. My mitigation was to limit the maximum fusion depth, which breaks large parameter kernels into smaller ones. But that is not guaranteed to work and not predictable. I can imagine that having a pass like this that allows more fine-grained controls might be necessary in some cases.
@echuraev FYI you can cap the fuse depth by https://github.com/apache/tvm/blob/720e7b1ebd9b789a1100dee7536d0633c7941dd1/tests/python/relay/test_pass_fuse_ops.py#L755
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] echuraev commented on a change in pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
echuraev commented on a change in pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#discussion_r660432231
##########
File path: src/relay/transforms/pattern_utils.h
##########
@@ -700,6 +700,13 @@ Expr StopFusion(Expr data);
Expr CastHint(Expr data, DataType dtype);
+inline Expr Concat(Expr x, int axis = 0) {
+ static const Op& op = Op::Get("concatenate");
+ auto attrs = make_object<ConcatenateAttrs>();
+ attrs->axis = axis;
+ return Call(op, {x}, Attrs(attrs), {});
+}
Review comment:
Done. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] echuraev commented on a change in pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
echuraev commented on a change in pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#discussion_r660432231
##########
File path: src/relay/transforms/pattern_utils.h
##########
@@ -700,6 +700,13 @@ Expr StopFusion(Expr data);
Expr CastHint(Expr data, DataType dtype);
+inline Expr Concat(Expr x, int axis = 0) {
+ static const Op& op = Op::Get("concatenate");
+ auto attrs = make_object<ConcatenateAttrs>();
+ attrs->axis = axis;
+ return Call(op, {x}, Attrs(attrs), {});
+}
Review comment:
Done. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] echuraev commented on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
echuraev commented on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-870453510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi edited a comment on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
masahi edited a comment on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-870050776
Not exactly, but I've dealt with a similar issue. My mitigation was to limit the maximum fusion depth, which breaks large parameter kernels into smaller ones. But that is not guaranteed to work and not predictable. I can imagine that having a pass like this that allows more fine-grained controls might be necessary in some cases.
@echuraev FYI you can cap the fuse depth by https://github.com/apache/tvm/blob/720e7b1ebd9b789a1100dee7536d0633c7941dd1/tests/python/relay/test_pass_fuse_ops.py#L755
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] mbrookhart commented on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
mbrookhart commented on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-869835330
cc @masahi
I remember Masa doing something like this for Vulkan at one point, but I'm not sure if that was a branch or if it ever got merged. If it's merged somewhere, maybe we should combine make this a generally available tool?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] mbrookhart commented on a change in pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
mbrookhart commented on a change in pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#discussion_r660126055
##########
File path: src/relay/transforms/pattern_utils.h
##########
@@ -700,6 +700,13 @@ Expr StopFusion(Expr data);
Expr CastHint(Expr data, DataType dtype);
+inline Expr Concat(Expr x, int axis = 0) {
+ static const Op& op = Op::Get("concatenate");
+ auto attrs = make_object<ConcatenateAttrs>();
+ attrs->axis = axis;
+ return Call(op, {x}, Attrs(attrs), {});
+}
Review comment:
Maybe just use the version here instead of duplicating? https://github.com/apache/tvm/blob/2915349458619deb5bd9b03670b33205938a8c02/src/relay/op/make_op.h#L45
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] echuraev commented on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
echuraev commented on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-870458257
> Minor nit.
>
> In general, I see the utility for single Relay kernels that have too many inputs, but I wonder if you'll hit this more for kernels post-fusion. This doesn't seem to tackle that for anything other than giant concatenation?
Yes, now it is solving only problem with concat layer. I thought that maybe this pass can be useful in case of split layer for example. But I wasn't able to reproduce the same problem for a split layer on several simple tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] jwfromm commented on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
jwfromm commented on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-869829030
@mbrookhart can you take a look at this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] mbrookhart commented on a change in pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
mbrookhart commented on a change in pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#discussion_r660126055
##########
File path: src/relay/transforms/pattern_utils.h
##########
@@ -700,6 +700,13 @@ Expr StopFusion(Expr data);
Expr CastHint(Expr data, DataType dtype);
+inline Expr Concat(Expr x, int axis = 0) {
+ static const Op& op = Op::Get("concatenate");
+ auto attrs = make_object<ConcatenateAttrs>();
+ attrs->axis = axis;
+ return Call(op, {x}, Attrs(attrs), {});
+}
Review comment:
Maybe just use the version here instead of duplicating? https://github.com/apache/tvm/blob/2915349458619deb5bd9b03670b33205938a8c02/src/relay/op/make_op.h#L45
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi merged pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
masahi merged pull request #8313:
URL: https://github.com/apache/tvm/pull/8313
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] mbrookhart edited a comment on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
mbrookhart edited a comment on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-870658270
I have no problems merging this as it stands, but I do think I have a bigger question:
Should we put some sort of logic into fusion to automatically stop fusion if the argument list grows too large per this setting? That should be more robust than arbitrarily limiting the fusion depth. It could of course be a second PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] mbrookhart commented on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
mbrookhart commented on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-869835330
cc @masahi
I remember Masa doing something like this for Vulkan at one point, but I'm not sure if that was a branch or if it ever got merged. If it's merged somewhere, maybe we should combine make this a generally available tool?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] echuraev commented on a change in pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
echuraev commented on a change in pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#discussion_r657054872
##########
File path: src/relay/transforms/split_args.cc
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file split_args.cc
+ */
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/transform.h>
+
+#include "pattern_utils.h"
+
+namespace tvm {
+namespace relay {
+
+class ArgumentSplitter : public ExprRewriter {
+ public:
+ explicit ArgumentSplitter(int max_function_args)
+ : max_function_args_(max_function_args), concat_op_(Op::Get("concatenate")) {}
+
+ Expr Rewrite_(const CallNode* call, const Expr& post) final {
+ if (max_function_args_ < 0) return post;
+ if (call->op == concat_op_) {
+ auto op = call->args[0].as<TupleNode>();
+ const auto param = call->attrs.as<ConcatenateAttrs>();
+ const int limit = max_function_args_ - 1; // one buffer with output
Review comment:
I have a doubt about this `- 1` here. But I didn't find how can I get the number of outputs from `relay::CallNode`. Maybe someone could help me, how can I get the number of outputs for the call in proper way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-870050776
Not exactly, but I've dealt with a similar issue. My mitigation was to limit the maximum fusion depth, which breaks large parameter kernels into smaller ones. But that is not guaranteed to work and not predictable. I can imagine that having a pass like this that allows more fine-grained controls might be necessary in some cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] echuraev commented on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
echuraev commented on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-870453510
> Not exactly, but I've dealt with a similar issue. My mitigation was to limit the maximum fusion depth, which breaks large parameter kernels into smaller ones. But that is not guaranteed to work and not predictable. I can imagine that having a pass like this that allows more fine-grained controls might be necessary in some cases.
>
> @echuraev FYI you can cap the fuse depth by
>
> https://github.com/apache/tvm/blob/720e7b1ebd9b789a1100dee7536d0633c7941dd1/tests/python/relay/test_pass_fuse_ops.py#L755
Thank you! I thought about the reducing fuse depth, but as you mentioned, it is not predictable and not guaranteed to work. This is why I think that this approach with splitting kernels is more robust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] echuraev commented on pull request #8313: [Metal] Add pass for splitting kernel with huge number of args
Posted by GitBox <gi...@apache.org>.
echuraev commented on pull request #8313:
URL: https://github.com/apache/tvm/pull/8313#issuecomment-870853665
> I have no problems merging this as it stands, but I do think I have a bigger question:
>
> Should we put some sort of logic into fusion to automatically stop fusion if the argument list grows too large per this setting? That should be more robust than arbitrarily limiting the fusion depth. It could of course be a second PR.
It's a good point and looks reasonable to add such logic into the fusion algorithm. It could help us to avoid some possible problems with number of arguments in the future. I think it would be better to do such logic in separate PR, due to problem with concat layer can appear and without fusing. For example, in the original problem we had many inputs because for each input to concat we had some preprocessing. And the fusing algorithm wasn't able to fuse these inputs into tuple due to this preprocessing. I don't think that the limit on the fusion depth can solve this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org