You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/11/24 23:52:14 UTC
[GitHub] [tvm] vegaluisjose opened a new pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
vegaluisjose opened a new pull request #6971:
URL: https://github.com/apache/tvm/pull/6971
# Integrating and simulating hardware accelerators in TVM
This RFC proposes support for integrating hardware designs in TVM,
specifically before they are manufactured in silicon or prototyped
in an FPGA. Regardless of the hardware target, ML accelerators
are built today, using the *waterfall* approach described in the
following [figure](https://github.com/vegaluisjose/figures/blob/main/tvm/rfc_verilator/today.png):
Starting from an accelerator idea, hardware is built incrementally
until is manufactured after several weeks or in most cases months of engineering.
Once this happens, software teams start working on integrating the new
piece of hardware to a particular ML framework of choice i.e., TVM.
One challenge of this approach is the fact that hardware-design decisions
are evaluated too late, which turns out to be counterproductive. Additionally,
there are higher probabilities that the target ML model gets updated
along the way, which makes critical the efficiency of this process.
Motivated by this reality, we believe ML accelerators should be integrated,
and more importantly evaluated once the hardware design process begins.
Concretely, this approach allows engineers to have design feedback
and hardware-software continuous integration, since the first day as
shown in the following [figure](https://github.com/vegaluisjose/figures/blob/main/tvm/rfc_verilator/idea.png):
We achieved this integration by leveraging the fact that most ML accelerators
are funnel down to a hardware language i.e., Verilog, regardless of the source
language. Moreover, we have today efficient Verilog-to-C++ open-source
compilers i.e., Verilator that can be used to interoperate via a simple
C interface (CFFI) with TVM.
This RFC proposes a simple codegen using BYOC, and an *opaque* kernel library, runtime,
and device interface that can be implemented by engineers to efficiently integrate
ML accelerators with TVM early in the design process.
Also, we provide a small example (demo) that shows how to offload an `add` instruction
from a Relay program to a simple hardware implementation (scalar adder) written in Verilog.
@tmoreau89 @liangfu @jroesch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tqchen edited a comment on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733932171
Re the hw related component, how about we put it into the https://github.com/apache/tvm-vta repo? Given that infrastructure are already setup for testing in the VTA component, so the code in the repo can also be tested against the latest version of tvm. We could make it as part of `vta/apps/hw-widgets`
I agree that having an RFC about the proposal would be really nice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
comaniac commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733913265
> @comaniac thank you for the feedback. The decision to add the Verilog sources in a 3rd party submodule came from TVM industry contributor concerns about leaving hardware sources out of the main repo for IP protection reasons. We are simply making sure all hardware sources are cleaved off from the main TVM repo.
I totally understand this concern and it makes sense to me. I completely agree with you that we should keep the hardware implementation out of the main repo. I was just concern the maintenance for this example. IMHO, it should also be fine if we bring the hardware example in and mention in the tutorial/blog post/comments that users are suggested to have a separate repo for this part. Anyways, I won't against the decision if everyone else are fine with it.
> I will move the Python test that I have in hw-widgets to follow what we have with other external codegens, making sure this work on our CI and does not bitrot.
Sounds like a plan. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] vegaluisjose commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
vegaluisjose commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733895668
> @vegaluisjose thanks for the explanation. It's now much more reasonable for me as I was thinking this is a general backend for everyone. In terms of being an example, this PR looks good to me except for the 3rd party dependency. To me, this is not the case as VTA, because VTA is a framework but hw-widgets is just an example. In addition, IIUC, with the 3rd party dependency, anyone who wants to contribute to this feature needs to first submit a PR to that repo, or it seems not possible to enable more ops such as Conv2D, otherwise.
>
> btw we may need a test case for this backend even it's just an example.
Regarding the test, I do have one located [here](https://github.com/octoml/hw-widgets/blob/main/scalar_add/tests/test_ops.py) that I hope we can figure how to eventually add it to CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tmoreau89 commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
tmoreau89 commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733895010
I agree on adding a test case - @vegaluisjose do you mind adding one to make sure this example doesn't bitrot?
I believe that the default CPU docker images are already build with Verilator, so hopefully we don't need to rebuild those.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tmoreau89 commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
tmoreau89 commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-735339748
Thank to @vegaluisjose and @comaniac @liangfu @zhiics @giuseros who reviewed the PR, it's now merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] giuseros commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
giuseros commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733910886
@vegaluisjose , it is ok for me too. Although, I would suggest to move the RFC on the discuss forum to get more visibility and comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tmoreau89 commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
tmoreau89 commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733942969
@tqchen if we go towards putting hardware sources under VTA, we may want to rename that repo to something like tvm-hardware or something similar under which apps and VTA sources would sit. And hopefully we'll see other designs in there as well. That's something we can do after an RFC discussion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tqchen commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733932171
Re the hw related component, how about we put it into the https://github.com/apache/tvm-vta repo? Given that infrastructure are already setup for testing in the VTA component. We could make it as part of `vta/apps/hw-widgets`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on a change in pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
comaniac commented on a change in pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#discussion_r530549060
##########
File path: src/relay/backend/contrib/verilator/codegen.cc
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/backend/contrib/verilator/codegen.cc
+ * \brief Implementation of Verilator codegen APIs.
+ */
+
+#include <tvm/relay/attrs/nn.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/transform.h>
+#include <tvm/relay/type.h>
+#include <tvm/runtime/module.h>
+#include <tvm/runtime/registry.h>
+
+#include <fstream>
+#include <numeric>
+#include <sstream>
+
+#include "../../../../runtime/contrib/json/json_node.h"
+#include "../../utils.h"
+#include "../codegen_json/codegen_json.h"
+
+namespace tvm {
+namespace relay {
+namespace contrib {
+
+using namespace backend;
+
+class VerilatorJSONSerializer : public backend::contrib::JSONSerializer {
+ using JSONGraphNode = tvm::runtime::json::JSONGraphNode;
+ using JSONGraphNodeEntry = tvm::runtime::json::JSONGraphNodeEntry;
+
+ public:
+ VerilatorJSONSerializer(const std::string& symbol, const Expr& expr)
+ : JSONSerializer(symbol, expr) {}
+
+ std::vector<JSONGraphNodeEntry> VisitExpr_(const CallNode* cn) override {
+ Expr expr = GetRef<Expr>(cn);
+ std::string name;
+ const CallNode* call = cn;
+ if (const auto* op_node = cn->op.as<OpNode>()) {
+ name = op_node->name;
+ } else {
+ LOG(FATAL) << "Verilator JSON runtime does not support calls to " << cn->op->GetTypeKey();
+ }
+
+ std::vector<JSONGraphNodeEntry> inputs;
+ for (const auto& arg : cn->args) {
+ auto res = VisitExpr(arg);
+ inputs.insert(inputs.end(), res.begin(), res.end());
+ }
+ auto node = std::make_shared<JSONGraphNode>(name, /* name_ */
+ "kernel", /* op_type_ */
+ inputs, 1 /* num_outputs_ */);
+ SetCallNodeAttribute(node, call);
+ return AddNode(node, GetRef<Expr>(cn));
Review comment:
This is too ad-hoc even you only support AddNode at this moment. It would be better to check the node type and throw an error when you find other unsupported nodes. This is also more flexible to add other nodes in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zhiics commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
zhiics commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733895030
@vegaluisjose thanks for the effort. I think this is a quite interesting implementation and yet another use case of BYOC. It is more like the json runtime example we had before but in the Verilator scenario. I think we need a test case. In addition, as @comaniac mentioned this would increase of maintenance (mostly from your side though) because we need to update both repos if there is change like adding operator support. Do you think it makes sense to contribute the kernels to TVM as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tmoreau89 commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
tmoreau89 commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733894434
@comaniac thank you for the feedback. The decision to add the Verilog sources in a 3rd party submodule came from TVM industry contributor concerns about leaving hardware sources out of the main repo for IP protection reasons. We are simply making sure all hardware sources are cleaved off from the main TVM repo.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] vegaluisjose edited a comment on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
vegaluisjose edited a comment on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733884979
Hi @comaniac , thank you for your feedback -- answers inline
> Thanks for this interesting idea and implementation! Hopefully we can get this in. Meanwhile, we should let more people review this PR especially it introduces a new 3rdparty dependency (`hw-widgets`). I also agree with @giuseros that it would be good to have an RFC talking about what you expect users to make use of this feature, because I didn't get this point after reviewing the PR.
>
> My confusion is what role "hw-widgets" plays in this feature.
>
> 1. According to its README, "These are examples of hardware designs for testing Verilator backend integration to TVM", is it just examples? If so then I don't think we really need an example to be in a 3rd party dependency. We should even bring the implementation into the TVM code base and only compile it when Verilator is enabled.
The `hw-widgets` repository is just an example on how to use this "backend" and a hardware codebase. The reason behind having a separate repo with examples is similar to the split that happen with VTA (3rdparty `vta-hw`).
> 2. What should I do if I have a new design in Verilator? Should I 1) compile my design to a library like you did for hw-widgets, 2) modify the `verilator_codegen` and `verilator_runtime` to link that library? If this is a case, then this PR is more like just an example instead of a framework.
I don't think I referred to this as a framework. I think this should be viewed more as another backend for TVM. Similarly to Ethos-N or DNNL, you need to have a precompiled library in your system but in this case this library is a cycle-accurate-version of your hardware as @giuseros pointed out. However, we need a separate ecosystem because like other external codegens, Verilator requires certain things to make it work.
For example, building floating-point hardware is far more complicated than integer or fixed-point hardware. Most of the backends in TVM, uses kernel libraries based on floating-point because hardware support is already free. But if you are building things from scratch, or if you are in a power-constraint environment i.e., edge, you are most likely to start with integer ops. Therefore, we need a mechanism to not only leverage full-blown-complex-accelerators but also the tiny-smallest-version-of-hardware that allow us to have a more vibrant *hardware ecosystem/community* compared to other frameworks. The other benefit of this is being capable of building hardware "incrementally" and evaluate your design-decisions with TVM over time.
> 3. The current implementation in this PR executes the graph op by op, but it seems not common for an accelerator. Is this just an example or your intention?
This backend can mirror what we currently have for DNNL in terms of single and merged operators (composite), no reason for stopping here. The reason why we have a single one is because we have only one example for this particular operator. Eventually, we are hoping for the community to join this effort and add examples that run more complicated use cases i.e., CONV2D, etc.
>
> cc @zhiics @tqchen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] vegaluisjose edited a comment on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
vegaluisjose edited a comment on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733843930
> Hi @vegaluisjose ,
>
> This looks very interesting! I would suggest to move the RFC discussion in https://discuss.tvm.apache.org/. Meanwhile, let me double check I understood the design. So, the idea is:
> a) Write Verilog
> b) Compile (e.g., with Verilator) in a C++ Cycle Accurate Model
> c) Link that model into TVM
> You are using BYOC so that you can split your graph on the operations you want, and offload the one you decided to support (potentially, the entire graph) to the C++ model. Is my understanding correct?
>
> Two questions:
> a) How hard is to open a route in TIR for this (to test smaller hw parts)? In theory `call_extern` should be sufficient?
> b) Why adding `hw-widgets` as a dependency? IIUC this is just an example, right?
Hi @giuseros ,
Yes to all of your points (a, b, c). This is an extension of a work we started exploring earlier this [year](https://homes.cs.washington.edu/~vegaluis/pdf/ieeemicro20_vega_lastlayer.pdf). The main difference between the paper and this RFC is the fact that we are allowing the hardware engineer to specify the level of detail of the simulating object. For example, if you have hardware code for simulating complex protocols i.e., AXI, then you could implement such protocol transactions using the proposed device interface. On the other hand, if you don't have such models or you want to bypass them to speed simulation then you can use hierarchical-paths to access memories or registers in your design, and control it the same way I am doing it in the current demo. The benefit of this *freedom* is that the interoperability should be independent of a particular hardware protocol or interface, and such interoperability should happen in the most productive way possible.
In terms of the other questions,
a) I haven't explored the TIR path yet. However, the benefit of doing such thing is leveraging other utilities available in the framework i.e., AutoTVM. In other words, BYOC could be considered a more shallow integration, but still useful.
b) The reason why we have the `hw-widgets` repo is that this stuff is hard to explain without a running example given that we are in a hardware-software boundary. I am working on a blogpost that will come after this gets merged that not only covers all of this but also shows how to offload operators from ML models. For example, we have already tested offloading `nn.bias_add()` to the same *scalar adder*, used in this example, from a quantized MobileNet model. As @liangfu said, this is a starting point for others.
Finally, this approach does not stop at ML accelerators only, you can imagine integrating a bare-metal processor out there in a similar fashion. There are plenty of RISCV processors out there using Verilator for simulation, so that could happen also. I am giving a small talk in TVM conf about this RFC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zhiics commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
zhiics commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733908298
@vegaluisjose sounds good to me. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tmoreau89 commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
tmoreau89 commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733919317
As @giuseros pointed out, an RFC about this work would be nice for longer term conversations. I'm hoping this PR can be merged soon for the TVM conf demo, but we should have longer-lasting discussion on extensions/applications of this work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] vegaluisjose commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
vegaluisjose commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733901174
Alright guys, sounds like we have a plan.
I will move the Python test that I have in `hw-widgets` to follow what we have with other [external codegens](https://github.com/apache/tvm/tree/main/tests/python/contrib), making sure this work on our CI and does not bitrot.
Does this sounds reasonable?
Thank you all for your valuable input @giuseros @comaniac @liangfu @tmoreau89 @zhiics
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zhiics commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
zhiics commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733895297
ahh, just saw @tmoreau89's comment on the separation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] vegaluisjose commented on a change in pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
vegaluisjose commented on a change in pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#discussion_r530584053
##########
File path: src/relay/backend/contrib/verilator/codegen.cc
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/backend/contrib/verilator/codegen.cc
+ * \brief Implementation of Verilator codegen APIs.
+ */
+
+#include <tvm/relay/attrs/nn.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/transform.h>
+#include <tvm/relay/type.h>
+#include <tvm/runtime/module.h>
+#include <tvm/runtime/registry.h>
+
+#include <fstream>
+#include <numeric>
+#include <sstream>
+
+#include "../../../../runtime/contrib/json/json_node.h"
+#include "../../utils.h"
+#include "../codegen_json/codegen_json.h"
+
+namespace tvm {
+namespace relay {
+namespace contrib {
+
+using namespace backend;
+
+class VerilatorJSONSerializer : public backend::contrib::JSONSerializer {
+ using JSONGraphNode = tvm::runtime::json::JSONGraphNode;
+ using JSONGraphNodeEntry = tvm::runtime::json::JSONGraphNodeEntry;
+
+ public:
+ VerilatorJSONSerializer(const std::string& symbol, const Expr& expr)
+ : JSONSerializer(symbol, expr) {}
+
+ std::vector<JSONGraphNodeEntry> VisitExpr_(const CallNode* cn) override {
+ Expr expr = GetRef<Expr>(cn);
+ std::string name;
+ const CallNode* call = cn;
+ if (const auto* op_node = cn->op.as<OpNode>()) {
+ name = op_node->name;
+ } else {
+ LOG(FATAL) << "Verilator JSON runtime does not support calls to " << cn->op->GetTypeKey();
+ }
+
+ std::vector<JSONGraphNodeEntry> inputs;
+ for (const auto& arg : cn->args) {
+ auto res = VisitExpr(arg);
+ inputs.insert(inputs.end(), res.begin(), res.end());
+ }
+ auto node = std::make_shared<JSONGraphNode>(name, /* name_ */
+ "kernel", /* op_type_ */
+ inputs, 1 /* num_outputs_ */);
+ SetCallNodeAttribute(node, call);
+ return AddNode(node, GetRef<Expr>(cn));
Review comment:
Great point, I was thinking the same. The thing is that we would need to add the function call in the verilator-runtime as well right?, leaving the freedom of marking the ops to be offloaded to the user (python). I was thinking on increasing this coverage as a follow-up PR, I did not want this to be too many lines to review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tmoreau89 merged pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
tmoreau89 merged pull request #6971:
URL: https://github.com/apache/tvm/pull/6971
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tmoreau89 commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
tmoreau89 commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-734968967
https://github.com/apache/tvm-vta/pull/18 has now been merged as requested by @tqchen .
Please update the submodule and paths to now point to tvm-vta, and bump the submodule version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] vegaluisjose commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
vegaluisjose commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733843930
> Hi @vegaluisjose ,
>
> This looks very interesting! I would suggest to move the RFC discussion in https://discuss.tvm.apache.org/. Meanwhile, let me double check I understood the design. So, the idea is:
> a) Write Verilog
> b) Compile (e.g., with Verilator) in a C++ Cycle Accurate Model
> c) Link that model into TVM
> You are using BYOC so that you can split your graph on the operations you want, and offload the one you decided to support (potentially, the entire graph) to the C++ model. Is my understanding correct?
>
> Two questions:
> a) How hard is to open a route in TIR for this (to test smaller hw parts)? In theory `call_extern` should be sufficient?
> b) Why adding `hw-widgets` as a dependency? IIUC this is just an example, right?
Hi @giuseros ,
Yes to all of your points (a, b, c). This is an extension of a work we started exploring earlier this [year](https://homes.cs.washington.edu/~vegaluis/pdf/ieeemicro20_vega_lastlayer.pdf). The main difference between the paper and this RFC is the fact that we are allowing the hardware engineer to specify the level of detail of the simulating object. For example, if you have hardware code for simulating complex protocols i.e., AXI, then you could implement such protocol transactions using the proposed device interface. On the other hand, if you don't have such models or you want to bypass them to speed simulation then you can use hierarchical-paths to access memories or registers in your design, and control it the same way I am doing it in the current demo. The benefit of this *freedom* is that the interoperability shouldn't depend on your protocol of choice, you can interact with your design in any way that is more productive for your use case.
In terms of the other questions,
a) I haven't explored the TIR path yet. However, the benefit of doing such thing is leveraging other utilities available in the framework i.e., AutoTVM. In other words, BYOC could be considered a more shallow integration, but still useful.
b) The reason why we have the `hw-widgets` repo is that this stuff is hard to explain without a running example given that we are in a hardware-software boundary. I am working on a blogpost that will come after this gets merged that not only covers all of this but also shows how to offload operators from ML models. For example, we have already tested offloading `nn.bias_add()` to the same *scalar adder*, used in this example, from a quantized MobileNet model. As @liangfu said, this is a starting point for others.
Finally, this approach does not stop at ML accelerators only, you can imagine integrating a bare-metal processor out there in a similar fashion. There are plenty of RISCV processors out there using Verilator for simulation, so that could happen also. I am giving a small talk in TVM conf about this RFC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tqchen edited a comment on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733932171
Re the hw related component, how about we put it into the https://github.com/apache/tvm-vta repo? Given that infrastructure are already setup for testing in the VTA component. We could make it as part of `vta/apps/hw-widgets`
I agree that having an RFC about the proposal would be really nice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] vegaluisjose commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
vegaluisjose commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733884979
Hi @comaniac , thank you for your feedback -- answers inline
> Thanks for this interesting idea and implementation! Hopefully we can get this in. Meanwhile, we should let more people review this PR especially it introduces a new 3rdparty dependency (`hw-widgets`). I also agree with @giuseros that it would be good to have an RFC talking about what you expect users to make use of this feature, because I didn't get this point after reviewing the PR.
>
> My confusion is what role "hw-widgets" plays in this feature.
>
> 1. According to its README, "These are examples of hardware designs for testing Verilator backend integration to TVM", is it just examples? If so then I don't think we really need an example to be in a 3rd party dependency. We should even bring the implementation into the TVM code base and only compile it when Verilator is enabled.
The `hw-widgets` repository is just an example on how to use this "backend" and a hardware codebase. The reason behind having a separate repo with examples is similar to the split that happen with VTA (3rdparty `vta-hw`).
> 2. What should I do if I have a new design in Verilator? Should I 1) compile my design to a library like you did for hw-widgets, 2) modify the `verilator_codegen` and `verilator_runtime` to link that library? If this is a case, then this PR is more like just an example instead of a framework.
I don't think I referred to this as a framework. I think this should be viewed more as another backend for TVM. Similarly to Ethos-N or DNNL, you need to have a precompiled library in your system but in this case this library is a cycle-accurate-version of your hardware as @giuseros pointed out. However, we need a separate ecosystem because like other external codegens, Verilator requires certain things to make it work.
For example, building floating-point hardware is far more complicated than integer or fixed-point hardware. Most of the backends in TVM, uses kernel libraries based on floating-point because hardware support is already free. But if you are building things from scratch, or if you are in a power-constraint environment i.e., edge, you are most likely to start with integer ops. Therefore, we need a mechanism to not only leverage full-blown-complex-accelerators but also the tiny-smallest-version-of-hardware that allow us to have a more vibrant *hardware ecosystem/community* compared to other frameworks. The other benefit of this is building hardware "incrementally" and evaluate your design-decisions with TVM over time.
> 3. The current implementation in this PR executes the graph op by op, but it seems not common for an accelerator. Is this just an example or your intention?
This backend can mirror what we currently have for DNNL in terms of single and merged operators (composite), no reason for stopping here. The reason why we have a single one is because we have only one example for this particular operator. Eventually, we are hoping for the community to join this effort and add examples that run more complicated use cases i.e., CONV2D, etc.
>
> cc @zhiics @tqchen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] giuseros commented on pull request #6971: [Hardware][Verilator] Integrating and simulating hardware accelerators in TVM
Posted by GitBox <gi...@apache.org>.
giuseros commented on pull request #6971:
URL: https://github.com/apache/tvm/pull/6971#issuecomment-733654476
Hi @vegaluisjose ,
This looks very interesting! I would suggest to move the RFC discussion in https://discuss.tvm.apache.org/. Meanwhile, let me double check I understood the design. So, the idea is:
a) Write Verilog
b) Compile (e.g., with Verilator) in a C++ Cycle Accurate Model
c) Link that model into TVM
You are using BYOC so that you can split your graph on the operations you want, and offload the one you decided to support (potentially, the entire graph) to the C++ model. Is my understanding correct?
Two questions:
a) How hard is to open a route in TIR for this (to test smaller hw parts)? In theory `call_extern` should be sufficient?
b) Why adding `hw-widgets` as a dependency? IIUC this is just an example, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org