You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/11/21 13:12:43 UTC
[GitHub] [tvm] krishnaraj36 opened a new pull request, #13450: [COLLAGE] Add more customization to support more targets
krishnaraj36 opened a new pull request, #13450:
URL: https://github.com/apache/tvm/pull/13450
1. Added custom cost module to provide a provision to incorporate custom cost estimator python function instead using default cost function.
eg: cost_estimator = CustomCostEstimator(py_fn_estimator="tvm.relay.collage.opencl_cost_estimator")
mod = CollagePartition(config, cost_estimator=cost_estimator)(mod)
2. Added provision to select BYOC fusion style for all compiler target.
eg : config = { "relay.collage.byoc_fusion_style": ["compiler.NoFusion", "compiler.TVMFusion"]}
ctxt = tvm.transform.PassContext(config=config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1065329229
##########
src/relay/collage/utils.cc:
##########
@@ -134,6 +134,19 @@ bool MustBeLowered(const Expr& expr) {
return false;
}
+std::vector<std::string> SplitString(std::string stmt, const char* del) {
Review Comment:
I have defined generic split string functions, which can be useful for any kind of splitting of string statement in future. So I defined as utility function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] srkreddy1238 commented on pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
srkreddy1238 commented on PR #13450:
URL: https://github.com/apache/tvm/pull/13450#issuecomment-1370495755
@tqchen Do we have moderator for Collage ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
masahi commented on PR #13450:
URL: https://github.com/apache/tvm/pull/13450#issuecomment-1371974855
Please remove typos and obvious grammar errors. I found many just looking over the code briefly. Please go through the change.
Also remove `TODO(mbs)` stuff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064827225
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
+ return DefaultBYOCFusionStyleForCompiler(compiler);
+ }
+ BYOCStyle byoc_fusion_style = kNoFusionBYOCStyle;
+ for (auto config_ : byoc_configs.value()) {
+ std::string byoc_str = static_cast<std::string>(config_);
+ std::string byoc_compiler = byoc_str.substr(0, byoc_str.find(".", 0));
+ if (byoc_compiler == compiler) {
+ std::string fusion_name = byoc_str.substr(byoc_str.find(".", 0) + 1, byoc_str.size());
Review Comment:
Its good idea, I have defined generic string split function in utils.cc file, we can call direct for this purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi merged pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
masahi merged PR #13450:
URL: https://github.com/apache/tvm/pull/13450
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on PR #13450:
URL: https://github.com/apache/tvm/pull/13450#issuecomment-1376794544
@masahi
Thanks for feedback, I have added the modification to the code as per your suggestion.
Please review and let me know your feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
masahi commented on PR #13450:
URL: https://github.com/apache/tvm/pull/13450#issuecomment-1371775072
Not sure and I'm not familiar with it either. But I can take care of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064284541
##########
src/relay/collage/custom_cost_estimator.h:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/collage/custom_cost_estimator.cc
+ * \brief A custom CostEstimator to support all different compiler cost.
+ */
+
+#ifndef TVM_RELAY_COLLAGE_CUSTOM_COST_ESTIMATOR_H_
+#define TVM_RELAY_COLLAGE_CUSTOM_COST_ESTIMATOR_H_
+
+#include <tvm/relay/function.h>
+
+#include "./cost.h"
+#include "./cost_estimator.h"
+
+namespace tvm {
+namespace relay {
+namespace collage {
+
+/*!
+ * \brief A custom cost estimator which can determine the cost of a candidate based on
+ * the candidate's target.
+ */
+class CustomCostEstimatorNode : public CostEstimatorNode {
+ public:
+ Cost Estimate(const IRModule& mod, const Target& target) const override;
+
+ static constexpr const char* _type_key = "relay.collage.CustomCostEstimator";
+ TVM_DECLARE_FINAL_OBJECT_INFO(CustomCostEstimatorNode, CostEstimatorNode);
+
+ protected:
+ /*!
+ * \brief Python implementation function register name to invoke.
Review Comment:
Corrected it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064846370
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
+ return DefaultBYOCFusionStyleForCompiler(compiler);
+ }
+ BYOCStyle byoc_fusion_style = kNoFusionBYOCStyle;
+ for (auto config_ : byoc_configs.value()) {
+ std::string byoc_str = static_cast<std::string>(config_);
+ std::string byoc_compiler = byoc_str.substr(0, byoc_str.find(".", 0));
+ if (byoc_compiler == compiler) {
+ std::string fusion_name = byoc_str.substr(byoc_str.find(".", 0) + 1, byoc_str.size());
+ if (fusion_name == "NoFusion") {
+ byoc_fusion_style = kNoFusionBYOCStyle;
+ } else if (fusion_name == "TVMFusion") {
+ byoc_fusion_style = kTVMFusionBYOCStyle;
+ } else if (fusion_name == "MaxDepthFusion") {
+ byoc_fusion_style = kArbitraryFusionBYOCStyle;
Review Comment:
Yes, it makes sense to have similar name, I have changed to "ArbitraryFusion".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064288369
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
Review Comment:
Corrected it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on PR #13450:
URL: https://github.com/apache/tvm/pull/13450#issuecomment-1373161563
> Please remove typos and obvious grammar errors. I found many just looking over the code briefly. Please go through your change.
>
> Also remove `TODO(mbs)` stuff.
@masahi
I have fixed the most to typo error and removed TODO (mbs), Please review and let me know your feedback.
If it looks good, please merge the branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on PR #13450:
URL: https://github.com/apache/tvm/pull/13450#issuecomment-1375940219
@masahi
Thank you for feedback.
I have added modification to code as per your suggestion and write out clarification for you few quarries.
Please review again and let me know you feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tvm-bot commented on pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
tvm-bot commented on PR #13450:
URL: https://github.com/apache/tvm/pull/13450#issuecomment-1322046372
<!---bot-comment-->
Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @-ing them in a comment.
<!--bot-comment-ccs-start-->
* No users to tag found in teams: `collage` <sub>See [#10317](https://github.com/apache/tvm/issues/10317) for details</sub><!--bot-comment-ccs-end-->
<sub>Generated by [tvm-bot](https://github.com/apache/tvm/blob/main/ci/README.md#github-actions)</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] srkreddy1238 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
srkreddy1238 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1046626218
##########
tests/python/contrib/test_clml/menangerie.py:
##########
@@ -0,0 +1,4179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
I see only resnet50, resnet50_16, mobilenet, mobilenet_16 are used for test cases.
You may remove other network definitions here.
##########
tests/python/contrib/test_clml/test_adreno_collage_targets.py:
##########
@@ -0,0 +1,400 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Compares Collage with various other baselines."""
+
+# CAUTION: Requires some changes in python/tvm/autotvm/task/dispatcher.py
Review Comment:
Is this caution relevant here ? i.e Do we need changes to AutoTVM ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1047230589
##########
tests/python/contrib/test_clml/test_adreno_collage_targets.py:
##########
@@ -0,0 +1,400 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Compares Collage with various other baselines."""
+
+# CAUTION: Requires some changes in python/tvm/autotvm/task/dispatcher.py
Review Comment:
Its not required, I have removed the line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1063893354
##########
src/relay/collage/custom_cost_estimator.cc:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/collage/custom_cost_estimator.cc
+ * \brief A custom CostEstimator to support Alternative cost functions.
Review Comment:
alternative
##########
src/relay/collage/custom_cost_estimator.h:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/collage/custom_cost_estimator.cc
+ * \brief A custom CostEstimator to support all different compiler cost.
+ */
+
+#ifndef TVM_RELAY_COLLAGE_CUSTOM_COST_ESTIMATOR_H_
+#define TVM_RELAY_COLLAGE_CUSTOM_COST_ESTIMATOR_H_
+
+#include <tvm/relay/function.h>
+
+#include "./cost.h"
+#include "./cost_estimator.h"
+
+namespace tvm {
+namespace relay {
+namespace collage {
+
+/*!
+ * \brief A custom cost estimator which can determine the cost of a candidate based on
+ * the candidate's target.
+ */
+class CustomCostEstimatorNode : public CostEstimatorNode {
+ public:
+ Cost Estimate(const IRModule& mod, const Target& target) const override;
+
+ static constexpr const char* _type_key = "relay.collage.CustomCostEstimator";
+ TVM_DECLARE_FINAL_OBJECT_INFO(CustomCostEstimatorNode, CostEstimatorNode);
+
+ protected:
+ /*!
+ * \brief Python implementation function register name to invoke.
Review Comment:
Please revise this sentence, it is broken.
##########
src/relay/collage/custom_cost_estimator.h:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/collage/custom_cost_estimator.cc
+ * \brief A custom CostEstimator to support all different compiler cost.
Review Comment:
to support target-specific cost functions
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
Review Comment:
Probably can remove `defined()`
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
Review Comment:
for the given compiler
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
+ return DefaultBYOCFusionStyleForCompiler(compiler);
+ }
+ BYOCStyle byoc_fusion_style = kNoFusionBYOCStyle;
+ for (auto config_ : byoc_configs.value()) {
+ std::string byoc_str = static_cast<std::string>(config_);
+ std::string byoc_compiler = byoc_str.substr(0, byoc_str.find(".", 0));
+ if (byoc_compiler == compiler) {
+ std::string fusion_name = byoc_str.substr(byoc_str.find(".", 0) + 1, byoc_str.size());
+ if (fusion_name == "NoFusion") {
+ byoc_fusion_style = kNoFusionBYOCStyle;
+ } else if (fusion_name == "TVMFusion") {
+ byoc_fusion_style = kTVMFusionBYOCStyle;
+ } else if (fusion_name == "MaxDepthFusion") {
+ byoc_fusion_style = kArbitraryFusionBYOCStyle;
Review Comment:
Why pick the name "MaxDepthFusion" over more obvious ones like "Arbitrary"?
##########
tests/python/contrib/test_clml/menangerie.py:
##########
@@ -0,0 +1,1527 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Use the models in python/tvm/relay/testing if possible
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
+ return DefaultBYOCFusionStyleForCompiler(compiler);
+ }
+ BYOCStyle byoc_fusion_style = kNoFusionBYOCStyle;
+ for (auto config_ : byoc_configs.value()) {
+ std::string byoc_str = static_cast<std::string>(config_);
+ std::string byoc_compiler = byoc_str.substr(0, byoc_str.find(".", 0));
+ if (byoc_compiler == compiler) {
+ std::string fusion_name = byoc_str.substr(byoc_str.find(".", 0) + 1, byoc_str.size());
Review Comment:
Two `substr` look cryptic without an example of an input string. I suggest making a function to do the split and call it like
auto [compiler_name, fusion_style] = ...
BTW, is there really a need to prefix the fusion style with the compiler name?
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
Review Comment:
unused
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
Review Comment:
Why is this an array? Isn't a single fusion style enough?
##########
src/relay/collage/custom_cost_estimator.h:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/collage/custom_cost_estimator.cc
+ * \brief A custom CostEstimator to support all different compiler cost.
+ */
+
+#ifndef TVM_RELAY_COLLAGE_CUSTOM_COST_ESTIMATOR_H_
+#define TVM_RELAY_COLLAGE_CUSTOM_COST_ESTIMATOR_H_
+
+#include <tvm/relay/function.h>
+
+#include "./cost.h"
+#include "./cost_estimator.h"
+
+namespace tvm {
+namespace relay {
+namespace collage {
+
+/*!
+ * \brief A custom cost estimator which can determine the cost of a candidate based on
Review Comment:
A cost estimator that uses a target-specific cost function
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
+ return DefaultBYOCFusionStyleForCompiler(compiler);
+ }
+ BYOCStyle byoc_fusion_style = kNoFusionBYOCStyle;
Review Comment:
I think it is better to use the output of `DefaultBYOCFusionStyleForCompiler(compiler)` here.
##########
tests/python/contrib/test_clml/test_adreno_collage_targets.py:
##########
@@ -0,0 +1,391 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Compares Collage with various other baselines."""
+
+import tvm
+import logging
+import tempfile
+import os
+import shutil
+import numpy as np
+from test_clml import menangerie
+from tvm import rpc
+from tvm.contrib import utils, ndk
+
+# The following are necessary to force global functions or pattern tables to be registered
+from tvm.relay.collage.collage import *
+from tvm.relay.op.contrib import clml
+import pytest
+
+logging.basicConfig(level=logging.INFO)
+
+
+########### Configuration ###########
+
+###
+### TVM Opencl AutoTvm log file name
+###
+TUNING_LOG = ""
+
+###
+### If true, run all models
+###
+ALL_MODELS = False
+
+###
+### If true, run all configurations
+###
+ALL_CONFIGS = False
+
+###
+### How aggressively to look for candidates?
+###
+TVM_MAX_DEPTH = 8
+BYOC_MAX_DEPTH = 8
+
+###
+### AutoTVM tuning parameters.
+###
+AUTOTVM_NUM_TRIALS = 1024
+AUTOTVM_EARLY_STOPPING = 600
+TIMEOUT = 10
+MEASURE_NUMBER = tvm.relay.collage.MEASURE_NUMBER
+MEASURE_REPEAT = tvm.relay.collage.MEASURE_REPEAT
+WARMUP_MIN_REPEAT_MS = tvm.relay.collage.WARMUP_MIN_REPEAT_MS
+
+##
+## RPC Build configuration
+##
+HOST = tvm.target.Target("llvm -mtriple=arm64-linux-android")
+OPENCL = tvm.target.Target("opencl", HOST)
+RPC_TRACKER_HOST = os.getenv("TVM_TRACKER_HOST", "localhost")
+RPC_TRACKER_PORT = int(os.getenv("TVM_TRACKER_PORT", 9090))
+RPC_KEY = os.getenv("RPC_DEVICE_KEY", "android")
+NDK_CROSS_COMPILER = os.getenv("TVM_NDK_CC", "aarch64-linux-android-g++")
+
+########### Runtime ###########
+
+# Code to run a model. The actual call to 'run' is appended at compile time.
+# We invoke the model as a sub-process so that we can wrap profiling tools around it.
+runner_template = f"""
+import tvm
+import tvm.runtime.vm
+import numpy as np
+import logging
+from tvm import rpc
+import os
+logging.basicConfig(level=logging.INFO)
+
+RPC_TRACKER_HOST = os.environ["TVM_TRACKER_HOST"]
+RPC_TRACKER_PORT = int(os.environ["TVM_TRACKER_PORT"])
+RPC_KEY = "android"
+MEASURE_NUMBER = {MEASURE_NUMBER}
+MEASURE_REPEAT = {MEASURE_REPEAT}
+WARMUP_MIN_REPEAT_MS = {WARMUP_MIN_REPEAT_MS}
+
+def arg_for(shape, dtype, device):
+ return tvm.nd.array(
+ np.random.rand(*shape).astype(dtype), device=device)
+
+def vm_estimate_seconds(device, vm, args):
+ vm.benchmark(device, repeat=1, number=1, min_repeat_ms=WARMUP_MIN_REPEAT_MS, **args)
+ return vm.benchmark(device, repeat=MEASURE_REPEAT, number=MEASURE_NUMBER, min_repeat_ms=0,
+ **args)
+
+
+def run(label, name, lib_path, input_shapes, input_dtypes):
+ logging.info(f"Loading compiled code for {{name}} generated by {{label}} from {{lib_path}}...")
+ tracker = rpc.connect_tracker(RPC_TRACKER_HOST, RPC_TRACKER_PORT)
+ remote = tracker.request(RPC_KEY, priority=0, session_timeout=600)
+ ctx = remote.cl(0)
+ remote_path = "/data/local/tmp/lib.so"
+ remote.upload(lib_path, target=remote_path)
+ lib = remote.load_module(remote_path)
+
+ vm_factory = tvm.runtime.vm.VirtualMachine(lib, ctx)
+ args = {{
+ input_name: arg_for(input_shapes[input_name], input_dtypes[input_name], ctx)
+ for input_name in input_shapes.keys()
+ }}
+ logging.info(f"Benchmarking for {{name}} generated by {{label}}...")
+ profile = vm_estimate_seconds(ctx, vm_factory, args)
+ logging.info(f"Benchmarked for {{name}} generated by {{label}}: {{profile}}")
+ logging.info(f"RESULT: {{label}} | {{name}} | {{profile.median * 1e3}}ms")
+
+if __name__ == "__main__":
+"""
+
+########### AutoTVM tuning helpers ###########
+
+
+def extract_autotvm_tasks(mod, target):
+ """Returns TVM kernels to tune for mod and target."""
+ return tvm.autotvm.task.extract_from_program(mod, target=target, params=None)
+
+
+def optional_tuning_records(log_filename):
+ """Returns existing tuning records, if any."""
+ if log_filename == "" or not os.path.exists(log_filename):
+ return tvm.autotvm.task.FallbackContext()
+ else:
+ return tvm.autotvm.task.ApplyHistoryBest(log_filename)
+
+
+def is_already_tuned(task, log_filename):
+ """Returns True if we already have a tuning record for task in turning logs in log_filename"""
+ if not os.path.exists(log_filename):
+ return False
+
+ dispatch_context = tvm.autotvm.task.ApplyHistoryBest(log_filename)
+ return dispatch_context._query_inside(task.target, task.workload)
+
+
+def tune_autotvm_tasks(tasks, log_filename):
+ """Appends to log filename the best strategies for tasks"""
+ if len(tasks) == 0:
+ return
+
+ measure_option = tvm.autotvm.measure_option(
+ builder=tvm.autotvm.LocalBuilder(build_func=ndk.create_shared, timeout=15),
+ runner=tvm.autotvm.RPCRunner(
+ RPC_KEY, host=RPC_TRACKER_HOST, port=RPC_TRACKER_PORT, number=100, timeout=15
+ ),
+ )
+
+ logging.info(
+ f"Using autotvm tuning for {len(tasks)} tasks with {AUTOTVM_NUM_TRIALS} trials, logging to {log_filename}"
+ )
+
+ # create tmp log file, starting with contents from existing log file
+ tmp_log_filename = log_filename + ".tmp"
+ if os.path.exists(tmp_log_filename):
+ os.remove(tmp_log_filename)
+ if os.path.exists(log_filename):
+ logging.info(f"Copying existing log {log_filename} to {tmp_log_filename}")
+ shutil.copy(log_filename, tmp_log_filename)
+
+ for i, task in enumerate(reversed(tasks)):
+ prefix = "[Task %2d/%2d] " % (i + 1, len(tasks))
+ logging.info(f"Considering task {task.name} {prefix}")
+ if is_already_tuned(task, tmp_log_filename):
+ logging.info(f"Re-using existing record for {task.name}")
+ continue
+
+ logging.info(f"Using autotvm to tune {task.name}")
+ tuner_obj = tvm.autotvm.tuner.XGBTuner(task, loss_type="rank")
+ if os.path.exists(tmp_log_filename):
+ tuner_obj.load_history(tvm.autotvm.record.load_from_file(tmp_log_filename))
+
+ # do tuning
+ n_trial = min(AUTOTVM_NUM_TRIALS, len(task.config_space))
+ tuner_obj.tune(
+ n_trial=n_trial,
+ early_stopping=AUTOTVM_EARLY_STOPPING,
+ measure_option=measure_option,
+ callbacks=[
+ tvm.autotvm.callback.progress_bar(n_trial, prefix=prefix),
+ tvm.autotvm.callback.log_to_file(tmp_log_filename),
+ ],
+ )
+
+ # Pick best records and copy back to main log file
+ tvm.autotvm.record.pick_best(tmp_log_filename, log_filename)
+ os.remove(tmp_log_filename)
+
+ logging.info("Done with autotvm tuning")
+
+
+def autotvm_tune_module(mod, target, log_filename):
+ if log_filename == "":
+ logging.info("Not tuning with autotvm since disabled")
+ return
+ # Extract and tune any TVM kernels. BYOC partitions will have no tasks extracted.
+ logging.info("Extracting tasks from overall module")
+ tasks = extract_autotvm_tasks(mod, target)
+ logging.info(f"Auto-tuning {len(tasks)} tasks from overall module")
+ tune_autotvm_tasks(tasks, log_filename)
+
+
+########### Drivers ###########
+
+
+def compile_and_benchmark(label, model, targets, tmp_dir):
+ """Compile model for target and run it with profiling."""
+ logging.info(f"Compiling {model['name']} using {label} with {targets}...")
+ model["mod"] = clml.preprocess_for_clml(model["mod"])
+ exe = tvm.relay.vm.compile(model["mod"], target=targets, params=model["params"])
+ lib = exe.mod
+ lib_path = os.path.join(tmp_dir, "lib.so")
+ logging.info(f"Exporting library to {lib_path}...")
+ lib.export_library(lib_path, cc=NDK_CROSS_COMPILER)
+ runner = f"{runner_template} run('{label}', '{model['name']}', '{lib_path}', {model['input_shapes']}, {model['input_dtypes']})\n"
+ runner_path = os.path.join(tmp_dir, "runner.py")
+ logging.info(f"Saving runner to {runner_path}...")
+ with open(runner_path, "w") as fo:
+ fo.write(runner)
+
+ logging.info(f"Invoking runner...")
+
+ os.system(f"python3 {runner_path}")
+
+
+# Custom cost function for Opencl RPC targets.
+@register_func("tvm.relay.collage.opencl_cost_estimator")
+def opencl_cost_estimator(mod, target):
+ mod = clml.preprocess_for_clml(mod) if "clml" == target.kind.name else mod
+ try:
+ # Build the module.
+ logging.info("Compiling module to estimate")
+ exe = tvm.relay.vm.compile(mod, target)
+ except RuntimeError as err:
+ # A build failure indicates the partition is not supported.
+ # eg trying to build an nn.batch_norm on GPU, which has no schedule since we assume it
+ # is only ever used with a tuple projection which is rewritten away.
+ logging.info("Assigning module infinite cost since unable to build: %s", err)
+ return math.inf
+
+ lib = exe.mod
+ tracker = rpc.connect_tracker(RPC_TRACKER_HOST, RPC_TRACKER_PORT)
+ remote = tracker.request(RPC_KEY, priority=0, session_timeout=600)
+ temp = utils.tempdir()
+ dso_binary = "dev_lib_cl.so"
+ dso_binary_path = temp.relpath(dso_binary)
+ ctx = remote.cl(0)
+ lib.export_library(dso_binary_path, cc=NDK_CROSS_COMPILER)
+ remote_path = dso_binary
+ remote.upload(dso_binary_path, target=remote_path)
+ lib = remote.load_module(remote_path)
+
+ vm_factory = tvm.runtime.vm.VirtualMachine(lib, ctx)
+ func_name = "main"
+ main_args = {v.name_hint: arg_for(v.checked_type, ctx) for v in mod[func_name].params}
+ cost = vm_factory.benchmark(
+ ctx, repeat=5, number=20, min_repeat_ms=0, func_name=func_name, **main_args
+ )
+ return cost.mean
+
+
+def collage(model):
+ """Run the Collage partitioner for a set of Opencl Adreno related targets and profile the result"""
+ logging.info(f"collage | {model['name']}")
+ logging.info("-------------- BEGIN ORIGINAL --------------")
+ logging.info(model["mod"])
+ logging.info("-------------- END ORIGINAL ----------------")
+ autotvm_tune_module(model["mod"], OPENCL, TUNING_LOG)
+ with optional_tuning_records(TUNING_LOG):
+ targets = []
+ targets.append(OPENCL)
+ use_fp16 = model["main_dtype"] == "float16"
+ tmp_dir = tempfile.mkdtemp()
+ targets.append(tvm.target.Target("clml", HOST))
+
+ # Register byoc fusion style for compiler with available
+ # options [compiler.NoFusion | compiler.TVMFusion | compiler.MaxDepthFusion]
+ config = {
+ "relay.collage.tvm_max_depth": TVM_MAX_DEPTH,
+ "relay.collage.byoc_max_depth": BYOC_MAX_DEPTH,
+ "relay.collage.byoc_fusion_style": ["clml.NoFusion"],
Review Comment:
Why can't you just pass "NoFusion" here? (without clml prefix)
##########
tests/python/contrib/test_clml/test_adreno_collage_targets.py:
##########
@@ -0,0 +1,391 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Compares Collage with various other baselines."""
+
+import tvm
+import logging
+import tempfile
+import os
+import shutil
+import numpy as np
+from test_clml import menangerie
+from tvm import rpc
+from tvm.contrib import utils, ndk
+
+# The following are necessary to force global functions or pattern tables to be registered
+from tvm.relay.collage.collage import *
+from tvm.relay.op.contrib import clml
+import pytest
+
+logging.basicConfig(level=logging.INFO)
+
+
+########### Configuration ###########
+
+###
+### TVM Opencl AutoTvm log file name
+###
+TUNING_LOG = ""
+
+###
+### If true, run all models
+###
+ALL_MODELS = False
+
+###
+### If true, run all configurations
+###
+ALL_CONFIGS = False
+
+###
+### How aggressively to look for candidates?
+###
+TVM_MAX_DEPTH = 8
+BYOC_MAX_DEPTH = 8
+
+###
+### AutoTVM tuning parameters.
+###
+AUTOTVM_NUM_TRIALS = 1024
+AUTOTVM_EARLY_STOPPING = 600
+TIMEOUT = 10
+MEASURE_NUMBER = tvm.relay.collage.MEASURE_NUMBER
+MEASURE_REPEAT = tvm.relay.collage.MEASURE_REPEAT
+WARMUP_MIN_REPEAT_MS = tvm.relay.collage.WARMUP_MIN_REPEAT_MS
+
+##
+## RPC Build configuration
+##
+HOST = tvm.target.Target("llvm -mtriple=arm64-linux-android")
+OPENCL = tvm.target.Target("opencl", HOST)
+RPC_TRACKER_HOST = os.getenv("TVM_TRACKER_HOST", "localhost")
+RPC_TRACKER_PORT = int(os.getenv("TVM_TRACKER_PORT", 9090))
+RPC_KEY = os.getenv("RPC_DEVICE_KEY", "android")
+NDK_CROSS_COMPILER = os.getenv("TVM_NDK_CC", "aarch64-linux-android-g++")
+
+########### Runtime ###########
+
+# Code to run a model. The actual call to 'run' is appended at compile time.
+# We invoke the model as a sub-process so that we can wrap profiling tools around it.
+runner_template = f"""
+import tvm
+import tvm.runtime.vm
+import numpy as np
+import logging
+from tvm import rpc
+import os
+logging.basicConfig(level=logging.INFO)
+
+RPC_TRACKER_HOST = os.environ["TVM_TRACKER_HOST"]
+RPC_TRACKER_PORT = int(os.environ["TVM_TRACKER_PORT"])
+RPC_KEY = "android"
+MEASURE_NUMBER = {MEASURE_NUMBER}
+MEASURE_REPEAT = {MEASURE_REPEAT}
+WARMUP_MIN_REPEAT_MS = {WARMUP_MIN_REPEAT_MS}
+
+def arg_for(shape, dtype, device):
+ return tvm.nd.array(
+ np.random.rand(*shape).astype(dtype), device=device)
+
+def vm_estimate_seconds(device, vm, args):
+ vm.benchmark(device, repeat=1, number=1, min_repeat_ms=WARMUP_MIN_REPEAT_MS, **args)
+ return vm.benchmark(device, repeat=MEASURE_REPEAT, number=MEASURE_NUMBER, min_repeat_ms=0,
+ **args)
+
+
+def run(label, name, lib_path, input_shapes, input_dtypes):
+ logging.info(f"Loading compiled code for {{name}} generated by {{label}} from {{lib_path}}...")
+ tracker = rpc.connect_tracker(RPC_TRACKER_HOST, RPC_TRACKER_PORT)
+ remote = tracker.request(RPC_KEY, priority=0, session_timeout=600)
+ ctx = remote.cl(0)
+ remote_path = "/data/local/tmp/lib.so"
+ remote.upload(lib_path, target=remote_path)
+ lib = remote.load_module(remote_path)
+
+ vm_factory = tvm.runtime.vm.VirtualMachine(lib, ctx)
+ args = {{
+ input_name: arg_for(input_shapes[input_name], input_dtypes[input_name], ctx)
+ for input_name in input_shapes.keys()
+ }}
+ logging.info(f"Benchmarking for {{name}} generated by {{label}}...")
+ profile = vm_estimate_seconds(ctx, vm_factory, args)
+ logging.info(f"Benchmarked for {{name}} generated by {{label}}: {{profile}}")
+ logging.info(f"RESULT: {{label}} | {{name}} | {{profile.median * 1e3}}ms")
+
+if __name__ == "__main__":
+"""
+
+########### AutoTVM tuning helpers ###########
+
+
+def extract_autotvm_tasks(mod, target):
+ """Returns TVM kernels to tune for mod and target."""
+ return tvm.autotvm.task.extract_from_program(mod, target=target, params=None)
+
+
+def optional_tuning_records(log_filename):
+ """Returns existing tuning records, if any."""
+ if log_filename == "" or not os.path.exists(log_filename):
+ return tvm.autotvm.task.FallbackContext()
+ else:
+ return tvm.autotvm.task.ApplyHistoryBest(log_filename)
+
+
+def is_already_tuned(task, log_filename):
+ """Returns True if we already have a tuning record for task in turning logs in log_filename"""
+ if not os.path.exists(log_filename):
+ return False
+
+ dispatch_context = tvm.autotvm.task.ApplyHistoryBest(log_filename)
+ return dispatch_context._query_inside(task.target, task.workload)
+
+
+def tune_autotvm_tasks(tasks, log_filename):
+ """Appends to log filename the best strategies for tasks"""
+ if len(tasks) == 0:
+ return
+
+ measure_option = tvm.autotvm.measure_option(
+ builder=tvm.autotvm.LocalBuilder(build_func=ndk.create_shared, timeout=15),
+ runner=tvm.autotvm.RPCRunner(
+ RPC_KEY, host=RPC_TRACKER_HOST, port=RPC_TRACKER_PORT, number=100, timeout=15
+ ),
+ )
+
+ logging.info(
+ f"Using autotvm tuning for {len(tasks)} tasks with {AUTOTVM_NUM_TRIALS} trials, logging to {log_filename}"
+ )
+
+ # create tmp log file, starting with contents from existing log file
+ tmp_log_filename = log_filename + ".tmp"
+ if os.path.exists(tmp_log_filename):
+ os.remove(tmp_log_filename)
+ if os.path.exists(log_filename):
+ logging.info(f"Copying existing log {log_filename} to {tmp_log_filename}")
+ shutil.copy(log_filename, tmp_log_filename)
+
+ for i, task in enumerate(reversed(tasks)):
+ prefix = "[Task %2d/%2d] " % (i + 1, len(tasks))
+ logging.info(f"Considering task {task.name} {prefix}")
+ if is_already_tuned(task, tmp_log_filename):
+ logging.info(f"Re-using existing record for {task.name}")
+ continue
+
+ logging.info(f"Using autotvm to tune {task.name}")
+ tuner_obj = tvm.autotvm.tuner.XGBTuner(task, loss_type="rank")
+ if os.path.exists(tmp_log_filename):
+ tuner_obj.load_history(tvm.autotvm.record.load_from_file(tmp_log_filename))
+
+ # do tuning
+ n_trial = min(AUTOTVM_NUM_TRIALS, len(task.config_space))
+ tuner_obj.tune(
+ n_trial=n_trial,
+ early_stopping=AUTOTVM_EARLY_STOPPING,
+ measure_option=measure_option,
+ callbacks=[
+ tvm.autotvm.callback.progress_bar(n_trial, prefix=prefix),
+ tvm.autotvm.callback.log_to_file(tmp_log_filename),
+ ],
+ )
+
+ # Pick best records and copy back to main log file
+ tvm.autotvm.record.pick_best(tmp_log_filename, log_filename)
+ os.remove(tmp_log_filename)
+
+ logging.info("Done with autotvm tuning")
+
+
+def autotvm_tune_module(mod, target, log_filename):
+ if log_filename == "":
+ logging.info("Not tuning with autotvm since disabled")
+ return
+ # Extract and tune any TVM kernels. BYOC partitions will have no tasks extracted.
+ logging.info("Extracting tasks from overall module")
+ tasks = extract_autotvm_tasks(mod, target)
+ logging.info(f"Auto-tuning {len(tasks)} tasks from overall module")
+ tune_autotvm_tasks(tasks, log_filename)
+
+
+########### Drivers ###########
+
+
+def compile_and_benchmark(label, model, targets, tmp_dir):
+ """Compile model for target and run it with profiling."""
+ logging.info(f"Compiling {model['name']} using {label} with {targets}...")
+ model["mod"] = clml.preprocess_for_clml(model["mod"])
+ exe = tvm.relay.vm.compile(model["mod"], target=targets, params=model["params"])
+ lib = exe.mod
+ lib_path = os.path.join(tmp_dir, "lib.so")
+ logging.info(f"Exporting library to {lib_path}...")
+ lib.export_library(lib_path, cc=NDK_CROSS_COMPILER)
+ runner = f"{runner_template} run('{label}', '{model['name']}', '{lib_path}', {model['input_shapes']}, {model['input_dtypes']})\n"
+ runner_path = os.path.join(tmp_dir, "runner.py")
+ logging.info(f"Saving runner to {runner_path}...")
+ with open(runner_path, "w") as fo:
+ fo.write(runner)
+
+ logging.info(f"Invoking runner...")
+
+ os.system(f"python3 {runner_path}")
Review Comment:
Why do you need this? Can't we wrap the boilarplate code in a function instead?
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
+ return DefaultBYOCFusionStyleForCompiler(compiler);
+ }
+ BYOCStyle byoc_fusion_style = kNoFusionBYOCStyle;
+ for (auto config_ : byoc_configs.value()) {
+ std::string byoc_str = static_cast<std::string>(config_);
Review Comment:
static_cast unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064834778
##########
tests/python/contrib/test_clml/test_adreno_collage_targets.py:
##########
@@ -0,0 +1,391 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Compares Collage with various other baselines."""
+
+import tvm
+import logging
+import tempfile
+import os
+import shutil
+import numpy as np
+from test_clml import menangerie
+from tvm import rpc
+from tvm.contrib import utils, ndk
+
+# The following are necessary to force global functions or pattern tables to be registered
+from tvm.relay.collage.collage import *
+from tvm.relay.op.contrib import clml
+import pytest
+
+logging.basicConfig(level=logging.INFO)
+
+
+########### Configuration ###########
+
+###
+### TVM Opencl AutoTvm log file name
+###
+TUNING_LOG = ""
+
+###
+### If true, run all models
+###
+ALL_MODELS = False
+
+###
+### If true, run all configurations
+###
+ALL_CONFIGS = False
+
+###
+### How aggressively to look for candidates?
+###
+TVM_MAX_DEPTH = 8
+BYOC_MAX_DEPTH = 8
+
+###
+### AutoTVM tuning parameters.
+###
+AUTOTVM_NUM_TRIALS = 1024
+AUTOTVM_EARLY_STOPPING = 600
+TIMEOUT = 10
+MEASURE_NUMBER = tvm.relay.collage.MEASURE_NUMBER
+MEASURE_REPEAT = tvm.relay.collage.MEASURE_REPEAT
+WARMUP_MIN_REPEAT_MS = tvm.relay.collage.WARMUP_MIN_REPEAT_MS
+
+##
+## RPC Build configuration
+##
+HOST = tvm.target.Target("llvm -mtriple=arm64-linux-android")
+OPENCL = tvm.target.Target("opencl", HOST)
+RPC_TRACKER_HOST = os.getenv("TVM_TRACKER_HOST", "localhost")
+RPC_TRACKER_PORT = int(os.getenv("TVM_TRACKER_PORT", 9090))
+RPC_KEY = os.getenv("RPC_DEVICE_KEY", "android")
+NDK_CROSS_COMPILER = os.getenv("TVM_NDK_CC", "aarch64-linux-android-g++")
+
+########### Runtime ###########
+
+# Code to run a model. The actual call to 'run' is appended at compile time.
+# We invoke the model as a sub-process so that we can wrap profiling tools around it.
+runner_template = f"""
+import tvm
+import tvm.runtime.vm
+import numpy as np
+import logging
+from tvm import rpc
+import os
+logging.basicConfig(level=logging.INFO)
+
+RPC_TRACKER_HOST = os.environ["TVM_TRACKER_HOST"]
+RPC_TRACKER_PORT = int(os.environ["TVM_TRACKER_PORT"])
+RPC_KEY = "android"
+MEASURE_NUMBER = {MEASURE_NUMBER}
+MEASURE_REPEAT = {MEASURE_REPEAT}
+WARMUP_MIN_REPEAT_MS = {WARMUP_MIN_REPEAT_MS}
+
+def arg_for(shape, dtype, device):
+ return tvm.nd.array(
+ np.random.rand(*shape).astype(dtype), device=device)
+
+def vm_estimate_seconds(device, vm, args):
+ vm.benchmark(device, repeat=1, number=1, min_repeat_ms=WARMUP_MIN_REPEAT_MS, **args)
+ return vm.benchmark(device, repeat=MEASURE_REPEAT, number=MEASURE_NUMBER, min_repeat_ms=0,
+ **args)
+
+
+def run(label, name, lib_path, input_shapes, input_dtypes):
+ logging.info(f"Loading compiled code for {{name}} generated by {{label}} from {{lib_path}}...")
+ tracker = rpc.connect_tracker(RPC_TRACKER_HOST, RPC_TRACKER_PORT)
+ remote = tracker.request(RPC_KEY, priority=0, session_timeout=600)
+ ctx = remote.cl(0)
+ remote_path = "/data/local/tmp/lib.so"
+ remote.upload(lib_path, target=remote_path)
+ lib = remote.load_module(remote_path)
+
+ vm_factory = tvm.runtime.vm.VirtualMachine(lib, ctx)
+ args = {{
+ input_name: arg_for(input_shapes[input_name], input_dtypes[input_name], ctx)
+ for input_name in input_shapes.keys()
+ }}
+ logging.info(f"Benchmarking for {{name}} generated by {{label}}...")
+ profile = vm_estimate_seconds(ctx, vm_factory, args)
+ logging.info(f"Benchmarked for {{name}} generated by {{label}}: {{profile}}")
+ logging.info(f"RESULT: {{label}} | {{name}} | {{profile.median * 1e3}}ms")
+
+if __name__ == "__main__":
+"""
+
+########### AutoTVM tuning helpers ###########
+
+
+def extract_autotvm_tasks(mod, target):
+ """Returns TVM kernels to tune for mod and target."""
+ return tvm.autotvm.task.extract_from_program(mod, target=target, params=None)
+
+
+def optional_tuning_records(log_filename):
+ """Returns existing tuning records, if any."""
+ if log_filename == "" or not os.path.exists(log_filename):
+ return tvm.autotvm.task.FallbackContext()
+ else:
+ return tvm.autotvm.task.ApplyHistoryBest(log_filename)
+
+
+def is_already_tuned(task, log_filename):
+ """Returns True if we already have a tuning record for task in turning logs in log_filename"""
+ if not os.path.exists(log_filename):
+ return False
+
+ dispatch_context = tvm.autotvm.task.ApplyHistoryBest(log_filename)
+ return dispatch_context._query_inside(task.target, task.workload)
+
+
+def tune_autotvm_tasks(tasks, log_filename):
+ """Appends to log filename the best strategies for tasks"""
+ if len(tasks) == 0:
+ return
+
+ measure_option = tvm.autotvm.measure_option(
+ builder=tvm.autotvm.LocalBuilder(build_func=ndk.create_shared, timeout=15),
+ runner=tvm.autotvm.RPCRunner(
+ RPC_KEY, host=RPC_TRACKER_HOST, port=RPC_TRACKER_PORT, number=100, timeout=15
+ ),
+ )
+
+ logging.info(
+ f"Using autotvm tuning for {len(tasks)} tasks with {AUTOTVM_NUM_TRIALS} trials, logging to {log_filename}"
+ )
+
+ # create tmp log file, starting with contents from existing log file
+ tmp_log_filename = log_filename + ".tmp"
+ if os.path.exists(tmp_log_filename):
+ os.remove(tmp_log_filename)
+ if os.path.exists(log_filename):
+ logging.info(f"Copying existing log {log_filename} to {tmp_log_filename}")
+ shutil.copy(log_filename, tmp_log_filename)
+
+ for i, task in enumerate(reversed(tasks)):
+ prefix = "[Task %2d/%2d] " % (i + 1, len(tasks))
+ logging.info(f"Considering task {task.name} {prefix}")
+ if is_already_tuned(task, tmp_log_filename):
+ logging.info(f"Re-using existing record for {task.name}")
+ continue
+
+ logging.info(f"Using autotvm to tune {task.name}")
+ tuner_obj = tvm.autotvm.tuner.XGBTuner(task, loss_type="rank")
+ if os.path.exists(tmp_log_filename):
+ tuner_obj.load_history(tvm.autotvm.record.load_from_file(tmp_log_filename))
+
+ # do tuning
+ n_trial = min(AUTOTVM_NUM_TRIALS, len(task.config_space))
+ tuner_obj.tune(
+ n_trial=n_trial,
+ early_stopping=AUTOTVM_EARLY_STOPPING,
+ measure_option=measure_option,
+ callbacks=[
+ tvm.autotvm.callback.progress_bar(n_trial, prefix=prefix),
+ tvm.autotvm.callback.log_to_file(tmp_log_filename),
+ ],
+ )
+
+ # Pick best records and copy back to main log file
+ tvm.autotvm.record.pick_best(tmp_log_filename, log_filename)
+ os.remove(tmp_log_filename)
+
+ logging.info("Done with autotvm tuning")
+
+
+def autotvm_tune_module(mod, target, log_filename):
+ if log_filename == "":
+ logging.info("Not tuning with autotvm since disabled")
+ return
+ # Extract and tune any TVM kernels. BYOC partitions will have no tasks extracted.
+ logging.info("Extracting tasks from overall module")
+ tasks = extract_autotvm_tasks(mod, target)
+ logging.info(f"Auto-tuning {len(tasks)} tasks from overall module")
+ tune_autotvm_tasks(tasks, log_filename)
+
+
+########### Drivers ###########
+
+
+def compile_and_benchmark(label, model, targets, tmp_dir):
+ """Compile model for target and run it with profiling."""
+ logging.info(f"Compiling {model['name']} using {label} with {targets}...")
+ model["mod"] = clml.preprocess_for_clml(model["mod"])
+ exe = tvm.relay.vm.compile(model["mod"], target=targets, params=model["params"])
+ lib = exe.mod
+ lib_path = os.path.join(tmp_dir, "lib.so")
+ logging.info(f"Exporting library to {lib_path}...")
+ lib.export_library(lib_path, cc=NDK_CROSS_COMPILER)
+ runner = f"{runner_template} run('{label}', '{model['name']}', '{lib_path}', {model['input_shapes']}, {model['input_dtypes']})\n"
+ runner_path = os.path.join(tmp_dir, "runner.py")
+ logging.info(f"Saving runner to {runner_path}...")
+ with open(runner_path, "w") as fo:
+ fo.write(runner)
+
+ logging.info(f"Invoking runner...")
+
+ os.system(f"python3 {runner_path}")
+
+
+# Custom cost function for Opencl RPC targets.
+@register_func("tvm.relay.collage.opencl_cost_estimator")
+def opencl_cost_estimator(mod, target):
+ mod = clml.preprocess_for_clml(mod) if "clml" == target.kind.name else mod
+ try:
+ # Build the module.
+ logging.info("Compiling module to estimate")
+ exe = tvm.relay.vm.compile(mod, target)
+ except RuntimeError as err:
+ # A build failure indicates the partition is not supported.
+ # eg trying to build an nn.batch_norm on GPU, which has no schedule since we assume it
+ # is only ever used with a tuple projection which is rewritten away.
+ logging.info("Assigning module infinite cost since unable to build: %s", err)
+ return math.inf
+
+ lib = exe.mod
+ tracker = rpc.connect_tracker(RPC_TRACKER_HOST, RPC_TRACKER_PORT)
+ remote = tracker.request(RPC_KEY, priority=0, session_timeout=600)
+ temp = utils.tempdir()
+ dso_binary = "dev_lib_cl.so"
+ dso_binary_path = temp.relpath(dso_binary)
+ ctx = remote.cl(0)
+ lib.export_library(dso_binary_path, cc=NDK_CROSS_COMPILER)
+ remote_path = dso_binary
+ remote.upload(dso_binary_path, target=remote_path)
+ lib = remote.load_module(remote_path)
+
+ vm_factory = tvm.runtime.vm.VirtualMachine(lib, ctx)
+ func_name = "main"
+ main_args = {v.name_hint: arg_for(v.checked_type, ctx) for v in mod[func_name].params}
+ cost = vm_factory.benchmark(
+ ctx, repeat=5, number=20, min_repeat_ms=0, func_name=func_name, **main_args
+ )
+ return cost.mean
+
+
+def collage(model):
+ """Run the Collage partitioner for a set of Opencl Adreno related targets and profile the result"""
+ logging.info(f"collage | {model['name']}")
+ logging.info("-------------- BEGIN ORIGINAL --------------")
+ logging.info(model["mod"])
+ logging.info("-------------- END ORIGINAL ----------------")
+ autotvm_tune_module(model["mod"], OPENCL, TUNING_LOG)
+ with optional_tuning_records(TUNING_LOG):
+ targets = []
+ targets.append(OPENCL)
+ use_fp16 = model["main_dtype"] == "float16"
+ tmp_dir = tempfile.mkdtemp()
+ targets.append(tvm.target.Target("clml", HOST))
+
+ # Register byoc fusion style for compiler with available
+ # options [compiler.NoFusion | compiler.TVMFusion | compiler.MaxDepthFusion]
+ config = {
+ "relay.collage.tvm_max_depth": TVM_MAX_DEPTH,
+ "relay.collage.byoc_max_depth": BYOC_MAX_DEPTH,
+ "relay.collage.byoc_fusion_style": ["clml.NoFusion"],
Review Comment:
Since collage can support multiple compiler target, to define each byoc compiler target fusion style, we have defined such that
e.g., In case of NVidia target -
byoc_fusion_style = [cutlass.NoFusion, cudnn.NoFusion, tensorrt.TVMFusion]
So, we have uniformly defined to support all such combinations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1065328299
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for given compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ BYOCStyle byoc_fusion_style = DefaultBYOCFusionStyleForCompiler(compiler);
+ if (!byoc_configs) {
+ LOG(INFO)<<"default fusion style";
+ return byoc_fusion_style;
+ }
+ for (auto config_ : byoc_configs.value()) {
+ std::vector<std::string> byoc_cfg = SplitString(config_, ".");
+ LOG(INFO) <<"byoc cfg: "<<byoc_cfg[0]<<" : "<<byoc_cfg[1];
+ if (byoc_cfg[0] == compiler) {
+ if (byoc_cfg[1] == "NoFusion") {
+ LOG(INFO)<<"NoFusion detected";
Review Comment:
Removed prints
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064284499
##########
src/relay/collage/custom_cost_estimator.h:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/collage/custom_cost_estimator.cc
+ * \brief A custom CostEstimator to support all different compiler cost.
Review Comment:
Corrected it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064288770
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
+ return DefaultBYOCFusionStyleForCompiler(compiler);
+ }
+ BYOCStyle byoc_fusion_style = kNoFusionBYOCStyle;
Review Comment:
There is no mandatory initialization required here, so removed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064846676
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
+ return DefaultBYOCFusionStyleForCompiler(compiler);
+ }
+ BYOCStyle byoc_fusion_style = kNoFusionBYOCStyle;
+ for (auto config_ : byoc_configs.value()) {
+ std::string byoc_str = static_cast<std::string>(config_);
Review Comment:
Yes, I have removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064843564
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
Review Comment:
Yes, it is redundant, I have removed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064824989
##########
tests/python/contrib/test_clml/menangerie.py:
##########
@@ -0,0 +1,1527 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Yes we can use existing networks from testing instead additional network file. I have removed the file and used network from testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064834394
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
+ return DefaultBYOCFusionStyleForCompiler(compiler);
+ }
+ BYOCStyle byoc_fusion_style = kNoFusionBYOCStyle;
+ for (auto config_ : byoc_configs.value()) {
+ std::string byoc_str = static_cast<std::string>(config_);
+ std::string byoc_compiler = byoc_str.substr(0, byoc_str.find(".", 0));
+ if (byoc_compiler == compiler) {
+ std::string fusion_name = byoc_str.substr(byoc_str.find(".", 0) + 1, byoc_str.size());
Review Comment:
Regarding compiler name prefix in fusion style config, since collage can support multiple compiler target, to define each byoc compiler target fusion style, we have defined such that
e.g. In case of NVidia target -
byoc_fusion_style = [cutlass.NoFusion, cudnn.NoFusion, tensorrt.TVMFusion]
So, we have uniformly defined to support all such combinations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064284629
##########
src/relay/collage/custom_cost_estimator.h:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/collage/custom_cost_estimator.cc
+ * \brief A custom CostEstimator to support all different compiler cost.
+ */
+
+#ifndef TVM_RELAY_COLLAGE_CUSTOM_COST_ESTIMATOR_H_
+#define TVM_RELAY_COLLAGE_CUSTOM_COST_ESTIMATOR_H_
+
+#include <tvm/relay/function.h>
+
+#include "./cost.h"
+#include "./cost_estimator.h"
+
+namespace tvm {
+namespace relay {
+namespace collage {
+
+/*!
+ * \brief A custom cost estimator which can determine the cost of a candidate based on
Review Comment:
Corrected it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1047229942
##########
tests/python/contrib/test_clml/menangerie.py:
##########
@@ -0,0 +1,4179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Removed the unused code!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] srkreddy1238 commented on pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
srkreddy1238 commented on PR #13450:
URL: https://github.com/apache/tvm/pull/13450#issuecomment-1337029288
@mbs-octoml can you take a look on this PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on PR #13450:
URL: https://github.com/apache/tvm/pull/13450#issuecomment-1371912072
> @masahi :
I have resolved all merge conflicts on this PR, can you please review and merge this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] srkreddy1238 commented on pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
srkreddy1238 commented on PR #13450:
URL: https://github.com/apache/tvm/pull/13450#issuecomment-1371767541
@masahi any moderator for Collage ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064843068
##########
tests/python/contrib/test_clml/test_adreno_collage_targets.py:
##########
@@ -0,0 +1,391 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Compares Collage with various other baselines."""
+
+import tvm
+import logging
+import tempfile
+import os
+import shutil
+import numpy as np
+from test_clml import menangerie
+from tvm import rpc
+from tvm.contrib import utils, ndk
+
+# The following are necessary to force global functions or pattern tables to be registered
+from tvm.relay.collage.collage import *
+from tvm.relay.op.contrib import clml
+import pytest
+
+logging.basicConfig(level=logging.INFO)
+
+
+########### Configuration ###########
+
+###
+### TVM Opencl AutoTvm log file name
+###
+TUNING_LOG = ""
+
+###
+### If true, run all models
+###
+ALL_MODELS = False
+
+###
+### If true, run all configurations
+###
+ALL_CONFIGS = False
+
+###
+### How aggressively to look for candidates?
+###
+TVM_MAX_DEPTH = 8
+BYOC_MAX_DEPTH = 8
+
+###
+### AutoTVM tuning parameters.
+###
+AUTOTVM_NUM_TRIALS = 1024
+AUTOTVM_EARLY_STOPPING = 600
+TIMEOUT = 10
+MEASURE_NUMBER = tvm.relay.collage.MEASURE_NUMBER
+MEASURE_REPEAT = tvm.relay.collage.MEASURE_REPEAT
+WARMUP_MIN_REPEAT_MS = tvm.relay.collage.WARMUP_MIN_REPEAT_MS
+
+##
+## RPC Build configuration
+##
+HOST = tvm.target.Target("llvm -mtriple=arm64-linux-android")
+OPENCL = tvm.target.Target("opencl", HOST)
+RPC_TRACKER_HOST = os.getenv("TVM_TRACKER_HOST", "localhost")
+RPC_TRACKER_PORT = int(os.getenv("TVM_TRACKER_PORT", 9090))
+RPC_KEY = os.getenv("RPC_DEVICE_KEY", "android")
+NDK_CROSS_COMPILER = os.getenv("TVM_NDK_CC", "aarch64-linux-android-g++")
+
+########### Runtime ###########
+
+# Code to run a model. The actual call to 'run' is appended at compile time.
+# We invoke the model as a sub-process so that we can wrap profiling tools around it.
+runner_template = f"""
+import tvm
+import tvm.runtime.vm
+import numpy as np
+import logging
+from tvm import rpc
+import os
+logging.basicConfig(level=logging.INFO)
+
+RPC_TRACKER_HOST = os.environ["TVM_TRACKER_HOST"]
+RPC_TRACKER_PORT = int(os.environ["TVM_TRACKER_PORT"])
+RPC_KEY = "android"
+MEASURE_NUMBER = {MEASURE_NUMBER}
+MEASURE_REPEAT = {MEASURE_REPEAT}
+WARMUP_MIN_REPEAT_MS = {WARMUP_MIN_REPEAT_MS}
+
+def arg_for(shape, dtype, device):
+ return tvm.nd.array(
+ np.random.rand(*shape).astype(dtype), device=device)
+
+def vm_estimate_seconds(device, vm, args):
+ vm.benchmark(device, repeat=1, number=1, min_repeat_ms=WARMUP_MIN_REPEAT_MS, **args)
+ return vm.benchmark(device, repeat=MEASURE_REPEAT, number=MEASURE_NUMBER, min_repeat_ms=0,
+ **args)
+
+
+def run(label, name, lib_path, input_shapes, input_dtypes):
+ logging.info(f"Loading compiled code for {{name}} generated by {{label}} from {{lib_path}}...")
+ tracker = rpc.connect_tracker(RPC_TRACKER_HOST, RPC_TRACKER_PORT)
+ remote = tracker.request(RPC_KEY, priority=0, session_timeout=600)
+ ctx = remote.cl(0)
+ remote_path = "/data/local/tmp/lib.so"
+ remote.upload(lib_path, target=remote_path)
+ lib = remote.load_module(remote_path)
+
+ vm_factory = tvm.runtime.vm.VirtualMachine(lib, ctx)
+ args = {{
+ input_name: arg_for(input_shapes[input_name], input_dtypes[input_name], ctx)
+ for input_name in input_shapes.keys()
+ }}
+ logging.info(f"Benchmarking for {{name}} generated by {{label}}...")
+ profile = vm_estimate_seconds(ctx, vm_factory, args)
+ logging.info(f"Benchmarked for {{name}} generated by {{label}}: {{profile}}")
+ logging.info(f"RESULT: {{label}} | {{name}} | {{profile.median * 1e3}}ms")
+
+if __name__ == "__main__":
+"""
+
+########### AutoTVM tuning helpers ###########
+
+
+def extract_autotvm_tasks(mod, target):
+ """Returns TVM kernels to tune for mod and target."""
+ return tvm.autotvm.task.extract_from_program(mod, target=target, params=None)
+
+
+def optional_tuning_records(log_filename):
+ """Returns existing tuning records, if any."""
+ if log_filename == "" or not os.path.exists(log_filename):
+ return tvm.autotvm.task.FallbackContext()
+ else:
+ return tvm.autotvm.task.ApplyHistoryBest(log_filename)
+
+
+def is_already_tuned(task, log_filename):
+ """Returns True if we already have a tuning record for task in turning logs in log_filename"""
+ if not os.path.exists(log_filename):
+ return False
+
+ dispatch_context = tvm.autotvm.task.ApplyHistoryBest(log_filename)
+ return dispatch_context._query_inside(task.target, task.workload)
+
+
+def tune_autotvm_tasks(tasks, log_filename):
+ """Appends to log filename the best strategies for tasks"""
+ if len(tasks) == 0:
+ return
+
+ measure_option = tvm.autotvm.measure_option(
+ builder=tvm.autotvm.LocalBuilder(build_func=ndk.create_shared, timeout=15),
+ runner=tvm.autotvm.RPCRunner(
+ RPC_KEY, host=RPC_TRACKER_HOST, port=RPC_TRACKER_PORT, number=100, timeout=15
+ ),
+ )
+
+ logging.info(
+ f"Using autotvm tuning for {len(tasks)} tasks with {AUTOTVM_NUM_TRIALS} trials, logging to {log_filename}"
+ )
+
+ # create tmp log file, starting with contents from existing log file
+ tmp_log_filename = log_filename + ".tmp"
+ if os.path.exists(tmp_log_filename):
+ os.remove(tmp_log_filename)
+ if os.path.exists(log_filename):
+ logging.info(f"Copying existing log {log_filename} to {tmp_log_filename}")
+ shutil.copy(log_filename, tmp_log_filename)
+
+ for i, task in enumerate(reversed(tasks)):
+ prefix = "[Task %2d/%2d] " % (i + 1, len(tasks))
+ logging.info(f"Considering task {task.name} {prefix}")
+ if is_already_tuned(task, tmp_log_filename):
+ logging.info(f"Re-using existing record for {task.name}")
+ continue
+
+ logging.info(f"Using autotvm to tune {task.name}")
+ tuner_obj = tvm.autotvm.tuner.XGBTuner(task, loss_type="rank")
+ if os.path.exists(tmp_log_filename):
+ tuner_obj.load_history(tvm.autotvm.record.load_from_file(tmp_log_filename))
+
+ # do tuning
+ n_trial = min(AUTOTVM_NUM_TRIALS, len(task.config_space))
+ tuner_obj.tune(
+ n_trial=n_trial,
+ early_stopping=AUTOTVM_EARLY_STOPPING,
+ measure_option=measure_option,
+ callbacks=[
+ tvm.autotvm.callback.progress_bar(n_trial, prefix=prefix),
+ tvm.autotvm.callback.log_to_file(tmp_log_filename),
+ ],
+ )
+
+ # Pick best records and copy back to main log file
+ tvm.autotvm.record.pick_best(tmp_log_filename, log_filename)
+ os.remove(tmp_log_filename)
+
+ logging.info("Done with autotvm tuning")
+
+
+def autotvm_tune_module(mod, target, log_filename):
+ if log_filename == "":
+ logging.info("Not tuning with autotvm since disabled")
+ return
+ # Extract and tune any TVM kernels. BYOC partitions will have no tasks extracted.
+ logging.info("Extracting tasks from overall module")
+ tasks = extract_autotvm_tasks(mod, target)
+ logging.info(f"Auto-tuning {len(tasks)} tasks from overall module")
+ tune_autotvm_tasks(tasks, log_filename)
+
+
+########### Drivers ###########
+
+
+def compile_and_benchmark(label, model, targets, tmp_dir):
+ """Compile model for target and run it with profiling."""
+ logging.info(f"Compiling {model['name']} using {label} with {targets}...")
+ model["mod"] = clml.preprocess_for_clml(model["mod"])
+ exe = tvm.relay.vm.compile(model["mod"], target=targets, params=model["params"])
+ lib = exe.mod
+ lib_path = os.path.join(tmp_dir, "lib.so")
+ logging.info(f"Exporting library to {lib_path}...")
+ lib.export_library(lib_path, cc=NDK_CROSS_COMPILER)
+ runner = f"{runner_template} run('{label}', '{model['name']}', '{lib_path}', {model['input_shapes']}, {model['input_dtypes']})\n"
+ runner_path = os.path.join(tmp_dir, "runner.py")
+ logging.info(f"Saving runner to {runner_path}...")
+ with open(runner_path, "w") as fo:
+ fo.write(runner)
+
+ logging.info(f"Invoking runner...")
+
+ os.system(f"python3 {runner_path}")
Review Comment:
Since collage doesn't generate tuning cache, so if we need collage tuned model for profiling or other use case, it required to tune again every time. So, we are making sure that tuned model saved with runner script that we can use for profiling or other use case.
I am working on to enable collage tunning cache generation, which will help to generate collage tunned quickly without all tuning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064848932
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
Review Comment:
Byoc fusion style config can have multiple compiler fusion styles.
e.g. In case of NVidia target -
byoc_fusion_style = [cutlass.NoFusion, cudnn.NoFusion, tensorrt.TVMFusion]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064843564
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ if (!byoc_configs.defined()) {
Review Comment:
Corrected it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1065377049
##########
tests/python/contrib/test_clml/test_adreno_collage_targets.py:
##########
@@ -0,0 +1,401 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Compares Collage with various other baselines."""
+
+import tvm
+import logging
+import tempfile
+import os
+import shutil
+import numpy as np
+from tvm.relay import testing
+from tvm import rpc
+from tvm.contrib import utils, ndk
+from tvm.relay.build_module import bind_params_by_name
+
+# The following are necessary to force global functions or pattern tables to be registered
+from tvm.relay.collage.collage import *
+from tvm.relay.op.contrib import clml
+import pytest
+
+logging.basicConfig(level=logging.INFO)
+
+
+########### Configuration ###########
+
+###
+### TVM Opencl AutoTvm log file name
+###
+TUNING_LOG = ""
+
+###
+### If true, run all models
+###
+ALL_MODELS = False
+
+###
+### If true, run all configurations
+###
+ALL_CONFIGS = False
+
+###
+### How aggressively to look for candidates?
+###
+TVM_MAX_DEPTH = 8
+BYOC_MAX_DEPTH = 8
+
+###
+### AutoTVM tuning parameters.
+###
+AUTOTVM_NUM_TRIALS = 1024
+AUTOTVM_EARLY_STOPPING = 600
+TIMEOUT = 10
+MEASURE_NUMBER = tvm.relay.collage.MEASURE_NUMBER
+MEASURE_REPEAT = tvm.relay.collage.MEASURE_REPEAT
+WARMUP_MIN_REPEAT_MS = tvm.relay.collage.WARMUP_MIN_REPEAT_MS
+
+##
+## RPC Build configuration
+##
+HOST = tvm.target.Target("llvm -mtriple=arm64-linux-android")
+OPENCL = tvm.target.Target("opencl", HOST)
+RPC_TRACKER_HOST = os.getenv("TVM_TRACKER_HOST", "localhost")
+RPC_TRACKER_PORT = int(os.getenv("TVM_TRACKER_PORT", 9090))
+RPC_KEY = os.getenv("RPC_DEVICE_KEY", "android")
+NDK_CROSS_COMPILER = os.getenv("TVM_NDK_CC", "aarch64-linux-android-g++")
+
+########### Runtime ###########
+
+# Code to run a model. The actual call to 'run' is appended at compile time.
+# We invoke the model as a sub-process so that we can wrap profiling tools around it.
+runner_template = f"""
Review Comment:
I had followed same test script used for Nvidia test case.(tests/python/relay/collage/demo_collage_partitioner.py)
Since here we are not doing any profiling for adreno target, so we can have single function instead runner template.
I have added the changes for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064277501
##########
src/relay/collage/custom_cost_estimator.cc:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/collage/custom_cost_estimator.cc
+ * \brief A custom CostEstimator to support Alternative cost functions.
Review Comment:
corrected typo error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
masahi commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1065125422
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for given compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ Optional<Array<String>> byoc_configs = ctxt->GetConfig(config_key, Optional<Array<String>>());
+ BYOCStyle byoc_fusion_style = DefaultBYOCFusionStyleForCompiler(compiler);
+ if (!byoc_configs) {
+ LOG(INFO)<<"default fusion style";
+ return byoc_fusion_style;
+ }
+ for (auto config_ : byoc_configs.value()) {
+ std::vector<std::string> byoc_cfg = SplitString(config_, ".");
+ LOG(INFO) <<"byoc cfg: "<<byoc_cfg[0]<<" : "<<byoc_cfg[1];
+ if (byoc_cfg[0] == compiler) {
+ if (byoc_cfg[1] == "NoFusion") {
+ LOG(INFO)<<"NoFusion detected";
Review Comment:
Remove `LOG(INFO)`, three in this file
##########
src/relay/collage/utils.cc:
##########
@@ -134,6 +134,19 @@ bool MustBeLowered(const Expr& expr) {
return false;
}
+std::vector<std::string> SplitString(std::string stmt, const char* del) {
Review Comment:
Just define this in `gather_partition_specs.cc` where it's used, since it is a one-off function
##########
tests/python/contrib/test_clml/test_adreno_collage_targets.py:
##########
@@ -0,0 +1,401 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Compares Collage with various other baselines."""
+
+import tvm
+import logging
+import tempfile
+import os
+import shutil
+import numpy as np
+from tvm.relay import testing
+from tvm import rpc
+from tvm.contrib import utils, ndk
+from tvm.relay.build_module import bind_params_by_name
+
+# The following are necessary to force global functions or pattern tables to be registered
+from tvm.relay.collage.collage import *
+from tvm.relay.op.contrib import clml
+import pytest
+
+logging.basicConfig(level=logging.INFO)
+
+
+########### Configuration ###########
+
+###
+### TVM Opencl AutoTvm log file name
+###
+TUNING_LOG = ""
+
+###
+### If true, run all models
+###
+ALL_MODELS = False
+
+###
+### If true, run all configurations
+###
+ALL_CONFIGS = False
+
+###
+### How aggressively to look for candidates?
+###
+TVM_MAX_DEPTH = 8
+BYOC_MAX_DEPTH = 8
+
+###
+### AutoTVM tuning parameters.
+###
+AUTOTVM_NUM_TRIALS = 1024
+AUTOTVM_EARLY_STOPPING = 600
+TIMEOUT = 10
+MEASURE_NUMBER = tvm.relay.collage.MEASURE_NUMBER
+MEASURE_REPEAT = tvm.relay.collage.MEASURE_REPEAT
+WARMUP_MIN_REPEAT_MS = tvm.relay.collage.WARMUP_MIN_REPEAT_MS
+
+##
+## RPC Build configuration
+##
+HOST = tvm.target.Target("llvm -mtriple=arm64-linux-android")
+OPENCL = tvm.target.Target("opencl", HOST)
+RPC_TRACKER_HOST = os.getenv("TVM_TRACKER_HOST", "localhost")
+RPC_TRACKER_PORT = int(os.getenv("TVM_TRACKER_PORT", 9090))
+RPC_KEY = os.getenv("RPC_DEVICE_KEY", "android")
+NDK_CROSS_COMPILER = os.getenv("TVM_NDK_CC", "aarch64-linux-android-g++")
+
+########### Runtime ###########
+
+# Code to run a model. The actual call to 'run' is appended at compile time.
+# We invoke the model as a sub-process so that we can wrap profiling tools around it.
+runner_template = f"""
Review Comment:
Just make it a function instead of a template.
##########
tests/python/contrib/test_clml/test_adreno_collage_targets.py:
##########
@@ -0,0 +1,391 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Compares Collage with various other baselines."""
+
+import tvm
+import logging
+import tempfile
+import os
+import shutil
+import numpy as np
+from test_clml import menangerie
+from tvm import rpc
+from tvm.contrib import utils, ndk
+
+# The following are necessary to force global functions or pattern tables to be registered
+from tvm.relay.collage.collage import *
+from tvm.relay.op.contrib import clml
+import pytest
+
+logging.basicConfig(level=logging.INFO)
+
+
+########### Configuration ###########
+
+###
+### TVM Opencl AutoTvm log file name
+###
+TUNING_LOG = ""
+
+###
+### If true, run all models
+###
+ALL_MODELS = False
+
+###
+### If true, run all configurations
+###
+ALL_CONFIGS = False
+
+###
+### How aggressively to look for candidates?
+###
+TVM_MAX_DEPTH = 8
+BYOC_MAX_DEPTH = 8
+
+###
+### AutoTVM tuning parameters.
+###
+AUTOTVM_NUM_TRIALS = 1024
+AUTOTVM_EARLY_STOPPING = 600
+TIMEOUT = 10
+MEASURE_NUMBER = tvm.relay.collage.MEASURE_NUMBER
+MEASURE_REPEAT = tvm.relay.collage.MEASURE_REPEAT
+WARMUP_MIN_REPEAT_MS = tvm.relay.collage.WARMUP_MIN_REPEAT_MS
+
+##
+## RPC Build configuration
+##
+HOST = tvm.target.Target("llvm -mtriple=arm64-linux-android")
+OPENCL = tvm.target.Target("opencl", HOST)
+RPC_TRACKER_HOST = os.getenv("TVM_TRACKER_HOST", "localhost")
+RPC_TRACKER_PORT = int(os.getenv("TVM_TRACKER_PORT", 9090))
+RPC_KEY = os.getenv("RPC_DEVICE_KEY", "android")
+NDK_CROSS_COMPILER = os.getenv("TVM_NDK_CC", "aarch64-linux-android-g++")
+
+########### Runtime ###########
+
+# Code to run a model. The actual call to 'run' is appended at compile time.
+# We invoke the model as a sub-process so that we can wrap profiling tools around it.
+runner_template = f"""
+import tvm
+import tvm.runtime.vm
+import numpy as np
+import logging
+from tvm import rpc
+import os
+logging.basicConfig(level=logging.INFO)
+
+RPC_TRACKER_HOST = os.environ["TVM_TRACKER_HOST"]
+RPC_TRACKER_PORT = int(os.environ["TVM_TRACKER_PORT"])
+RPC_KEY = "android"
+MEASURE_NUMBER = {MEASURE_NUMBER}
+MEASURE_REPEAT = {MEASURE_REPEAT}
+WARMUP_MIN_REPEAT_MS = {WARMUP_MIN_REPEAT_MS}
+
+def arg_for(shape, dtype, device):
+ return tvm.nd.array(
+ np.random.rand(*shape).astype(dtype), device=device)
+
+def vm_estimate_seconds(device, vm, args):
+ vm.benchmark(device, repeat=1, number=1, min_repeat_ms=WARMUP_MIN_REPEAT_MS, **args)
+ return vm.benchmark(device, repeat=MEASURE_REPEAT, number=MEASURE_NUMBER, min_repeat_ms=0,
+ **args)
+
+
+def run(label, name, lib_path, input_shapes, input_dtypes):
+ logging.info(f"Loading compiled code for {{name}} generated by {{label}} from {{lib_path}}...")
+ tracker = rpc.connect_tracker(RPC_TRACKER_HOST, RPC_TRACKER_PORT)
+ remote = tracker.request(RPC_KEY, priority=0, session_timeout=600)
+ ctx = remote.cl(0)
+ remote_path = "/data/local/tmp/lib.so"
+ remote.upload(lib_path, target=remote_path)
+ lib = remote.load_module(remote_path)
+
+ vm_factory = tvm.runtime.vm.VirtualMachine(lib, ctx)
+ args = {{
+ input_name: arg_for(input_shapes[input_name], input_dtypes[input_name], ctx)
+ for input_name in input_shapes.keys()
+ }}
+ logging.info(f"Benchmarking for {{name}} generated by {{label}}...")
+ profile = vm_estimate_seconds(ctx, vm_factory, args)
+ logging.info(f"Benchmarked for {{name}} generated by {{label}}: {{profile}}")
+ logging.info(f"RESULT: {{label}} | {{name}} | {{profile.median * 1e3}}ms")
+
+if __name__ == "__main__":
+"""
+
+########### AutoTVM tuning helpers ###########
+
+
+def extract_autotvm_tasks(mod, target):
+ """Returns TVM kernels to tune for mod and target."""
+ return tvm.autotvm.task.extract_from_program(mod, target=target, params=None)
+
+
+def optional_tuning_records(log_filename):
+ """Returns existing tuning records, if any."""
+ if log_filename == "" or not os.path.exists(log_filename):
+ return tvm.autotvm.task.FallbackContext()
+ else:
+ return tvm.autotvm.task.ApplyHistoryBest(log_filename)
+
+
+def is_already_tuned(task, log_filename):
+ """Returns True if we already have a tuning record for task in turning logs in log_filename"""
+ if not os.path.exists(log_filename):
+ return False
+
+ dispatch_context = tvm.autotvm.task.ApplyHistoryBest(log_filename)
+ return dispatch_context._query_inside(task.target, task.workload)
+
+
+def tune_autotvm_tasks(tasks, log_filename):
+ """Appends to log filename the best strategies for tasks"""
+ if len(tasks) == 0:
+ return
+
+ measure_option = tvm.autotvm.measure_option(
+ builder=tvm.autotvm.LocalBuilder(build_func=ndk.create_shared, timeout=15),
+ runner=tvm.autotvm.RPCRunner(
+ RPC_KEY, host=RPC_TRACKER_HOST, port=RPC_TRACKER_PORT, number=100, timeout=15
+ ),
+ )
+
+ logging.info(
+ f"Using autotvm tuning for {len(tasks)} tasks with {AUTOTVM_NUM_TRIALS} trials, logging to {log_filename}"
+ )
+
+ # create tmp log file, starting with contents from existing log file
+ tmp_log_filename = log_filename + ".tmp"
+ if os.path.exists(tmp_log_filename):
+ os.remove(tmp_log_filename)
+ if os.path.exists(log_filename):
+ logging.info(f"Copying existing log {log_filename} to {tmp_log_filename}")
+ shutil.copy(log_filename, tmp_log_filename)
+
+ for i, task in enumerate(reversed(tasks)):
+ prefix = "[Task %2d/%2d] " % (i + 1, len(tasks))
+ logging.info(f"Considering task {task.name} {prefix}")
+ if is_already_tuned(task, tmp_log_filename):
+ logging.info(f"Re-using existing record for {task.name}")
+ continue
+
+ logging.info(f"Using autotvm to tune {task.name}")
+ tuner_obj = tvm.autotvm.tuner.XGBTuner(task, loss_type="rank")
+ if os.path.exists(tmp_log_filename):
+ tuner_obj.load_history(tvm.autotvm.record.load_from_file(tmp_log_filename))
+
+ # do tuning
+ n_trial = min(AUTOTVM_NUM_TRIALS, len(task.config_space))
+ tuner_obj.tune(
+ n_trial=n_trial,
+ early_stopping=AUTOTVM_EARLY_STOPPING,
+ measure_option=measure_option,
+ callbacks=[
+ tvm.autotvm.callback.progress_bar(n_trial, prefix=prefix),
+ tvm.autotvm.callback.log_to_file(tmp_log_filename),
+ ],
+ )
+
+ # Pick best records and copy back to main log file
+ tvm.autotvm.record.pick_best(tmp_log_filename, log_filename)
+ os.remove(tmp_log_filename)
+
+ logging.info("Done with autotvm tuning")
+
+
+def autotvm_tune_module(mod, target, log_filename):
+ if log_filename == "":
+ logging.info("Not tuning with autotvm since disabled")
+ return
+ # Extract and tune any TVM kernels. BYOC partitions will have no tasks extracted.
+ logging.info("Extracting tasks from overall module")
+ tasks = extract_autotvm_tasks(mod, target)
+ logging.info(f"Auto-tuning {len(tasks)} tasks from overall module")
+ tune_autotvm_tasks(tasks, log_filename)
+
+
+########### Drivers ###########
+
+
+def compile_and_benchmark(label, model, targets, tmp_dir):
+ """Compile model for target and run it with profiling."""
+ logging.info(f"Compiling {model['name']} using {label} with {targets}...")
+ model["mod"] = clml.preprocess_for_clml(model["mod"])
+ exe = tvm.relay.vm.compile(model["mod"], target=targets, params=model["params"])
+ lib = exe.mod
+ lib_path = os.path.join(tmp_dir, "lib.so")
+ logging.info(f"Exporting library to {lib_path}...")
+ lib.export_library(lib_path, cc=NDK_CROSS_COMPILER)
+ runner = f"{runner_template} run('{label}', '{model['name']}', '{lib_path}', {model['input_shapes']}, {model['input_dtypes']})\n"
+ runner_path = os.path.join(tmp_dir, "runner.py")
+ logging.info(f"Saving runner to {runner_path}...")
+ with open(runner_path, "w") as fo:
+ fo.write(runner)
+
+ logging.info(f"Invoking runner...")
+
+ os.system(f"python3 {runner_path}")
Review Comment:
That doesn't answer my question. Why not, instead of `runner_template` thing and invoke the string with `os.system`, wrap the same functionality in a normal function and call it however time you want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] krishnaraj36 commented on a diff in pull request #13450: [COLLAGE] Add more customization to support more targets
Posted by GitBox <gi...@apache.org>.
krishnaraj36 commented on code in PR #13450:
URL: https://github.com/apache/tvm/pull/13450#discussion_r1064288283
##########
src/relay/collage/gather_partition_specs.cc:
##########
@@ -103,6 +101,38 @@ BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
}
}
+/*!
+ * \brief Returns the fusion style for /p compiler.
+ */
+BYOCStyle BYOCFusionStyleForCompiler(const String& compiler) {
+ tvm::transform::PassContext ctxt = tvm::transform::PassContext::Current();
+ std::string config_key = "relay.collage.byoc_fusion_style";
+ std::string compiler_id = compiler;
Review Comment:
Removed unused code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org