You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/02/03 11:25:26 UTC

[GitHub] [tvm] t-vi opened a new pull request #7401: WIP/RFC: initial stab at TorchScript fallback

t-vi opened a new pull request #7401:
URL: https://github.com/apache/tvm/pull/7401


   This patch adds a support for calling TorchScript. This can be used fallback for when torch operators are not yet implemented or if one wants to incorporate bespoke PyTorch custom ops into TVM with ease.
   
   It adds
   - a new relay `torchop` that takes a variable number of inputs and executes a provided TorchScript (aka PyTorch JIT) function,
   - a backend module class that calls into LibTorch (aka the C++ bindings underlying PyTorch), executing a TorchScript function.
   
   The key addition to the relay infrastructure is that while leaving `num_inputs == -1` on operator registration is documented to indicate a variable number of inputs, the type inference pass is not prepared to deal with it and instead requires the number of arguments provided to match the number of arguments with `add_argument` on operator registration. The proposed change to the type inference is then to match the declared arguments but to allow additional arguments if `num_inputs` is `-1`.
   
   The other detail is that it uses a string attribute in the call node's attributes to take the serialized TorchScript. This is a bit fishy as the serialized representation is binary. I used serialization to get TorchScript into TVM at the C++ level as it is tricky to interoperate between PyBind-wrapped objects (TorchScript in PyTorch) and the TVM FFI, but we might pass things around as handles later (but I'm not sure if that works well with attributes). I would be glad to have your advice on this.
   
   Currently I only support one output tensor and FP32, but that is straightforward to make flexible, and I would do this in parallel to the more fundamental discussion e.g. around the things above.
   
   Even though this is in draft state, I'm opening this PR to make discussions concrete in terms of code.
   
   I'm also posting this to the forums and will link here.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1061502358


   Great! Can you remove `WIP` from the title now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on a change in pull request #7401: RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on a change in pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#discussion_r822602114



##########
File path: src/runtime/contrib/libtorch/libtorch_runtime.cc
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/runtime/contrib/libtorch/libtorch_runtime.cc
+ * \brief runtime implementation for LibTorch/TorchScript.
+ */
+
+// we do not want clang to reorder our includes
+// clang-format off
+#include <tvm/runtime/module.h>
+#include <tvm/runtime/ndarray.h>
+#include <tvm/runtime/registry.h>
+
+#include <ATen/dlpack.h>
+#include <ATen/DLConvertor.h>
+#include <torch/csrc/jit/serialization/import.h>
+#include <torch/torch.h>
+
+// clang-format on
+
+#include <cstddef>
+#include <string>
+#include <tuple>
+#include <type_traits>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+namespace tvm {
+namespace runtime {
+namespace contrib {
+
+static void monly_deleter(DLManagedTensor* self) { delete self; }
+
+void run_torch_module(torch::jit::Module* module, TVMArgs args, TVMRetValue* rv) {
+  std::vector<torch::jit::IValue> inputs;
+  std::vector<torch::Tensor> outputs;
+  auto m = module->get_method("forward");
+  for (int i = 0; i < args.size(); i++) {
+    const DLTensor* arg;
+    if (args[i].IsObjectRef<NDArray>()) {
+      NDArray arr = args[i];
+      arg = arr.operator->();
+    } else {
+      arg = args[i].operator DLTensor*();
+    }
+    DLManagedTensor* inp = new DLManagedTensor{};
+    inp->dl_tensor = *arg;
+    inp->deleter = &monly_deleter;
+    // m.num_inputs includes the self argument of forward(self, ...)
+    // num_inputs - 1 is the number of (Tensor) inputs
+    if (i < static_cast<int>(m.num_inputs()) - 1) {
+      inputs.emplace_back(at::fromDLPack(inp));
+    } else {
+      outputs.emplace_back(at::fromDLPack(inp));
+    }
+  }
+  ICHECK(outputs.size() == 1) << "wrong number of args, can handle only one output";
+  torch::Tensor res = module->forward(inputs).toTensor();
+  outputs[0].copy_(res);  // too bad

Review comment:
       So I fixed your other two comments. I would think that for this, it might be best to see if we can do zero copy later.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-773154955


   > I'm curious how it integrates with PyTorch frontend. Do we convert every op not supported to relay.torchop, run BYOC flow to get TorchScript subgraphs, and send them to libtorch? Sounds interesting!
   
   This is how I'd like it to work out. I've been thinking what the best "level" is and while the operator level might seem attractive, but I I'm not sure there is an easy way to run individual operators.
   It won't help with out favourite inplace problems, though.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on a change in pull request #7401: RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on a change in pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#discussion_r822559313



##########
File path: src/runtime/contrib/libtorch/libtorch_runtime.cc
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/runtime/contrib/libtorch/libtorch_runtime.cc
+ * \brief runtime implementation for LibTorch/TorchScript.
+ */
+
+// we do not want clang to reorder our includes
+// clang-format off
+#include <tvm/runtime/module.h>
+#include <tvm/runtime/ndarray.h>
+#include <tvm/runtime/registry.h>
+
+#include <ATen/dlpack.h>
+#include <ATen/DLConvertor.h>
+#include <torch/csrc/jit/serialization/import.h>
+#include <torch/torch.h>
+
+// clang-format on
+
+#include <cstddef>
+#include <string>
+#include <tuple>
+#include <type_traits>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+namespace tvm {
+namespace runtime {
+namespace contrib {
+
+static void monly_deleter(DLManagedTensor* self) { delete self; }
+
+void run_torch_module(torch::jit::Module* module, TVMArgs args, TVMRetValue* rv) {
+  std::vector<torch::jit::IValue> inputs;
+  std::vector<torch::Tensor> outputs;
+  auto m = module->get_method("forward");
+  for (int i = 0; i < args.size(); i++) {
+    const DLTensor* arg;
+    if (args[i].IsObjectRef<NDArray>()) {
+      NDArray arr = args[i];
+      arg = arr.operator->();
+    } else {
+      arg = args[i].operator DLTensor*();
+    }
+    DLManagedTensor* inp = new DLManagedTensor{};
+    inp->dl_tensor = *arg;
+    inp->deleter = &monly_deleter;
+    // m.num_inputs includes the self argument of forward(self, ...)
+    // num_inputs - 1 is the number of (Tensor) inputs
+    if (i < static_cast<int>(m.num_inputs()) - 1) {
+      inputs.emplace_back(at::fromDLPack(inp));
+    } else {
+      outputs.emplace_back(at::fromDLPack(inp));
+    }
+  }
+  ICHECK(outputs.size() == 1) << "wrong number of args, can handle only one output";
+  torch::Tensor res = module->forward(inputs).toTensor();
+  outputs[0].copy_(res);  // too bad

Review comment:
       I could use a hint how to get the graph executor here, though.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-919961143


   So I have been mulling over the best granularity / approach. Currently I'm taking TorchScript functions / graphs as the unit I'm working with. An alternative could be to move to the PyTorch operator level (so one aten::...-call) - which would seem to be more natural in Relay - but then one would either have separate TVM Modules for each operator or have to piece things back together into a graph.
   
   What do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] PhilippvK commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
PhilippvK commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-846545791


   Hey @t-vi, the idea of a fallback for unsupported TorchScript Ops is great. I am currently pursuing a similar approach for unsupported (and custom) TFLite Ops.
   
   I also stumbled over the issue that  `num_inpust == -1` leads to problems in the `type_infer` step and "solved" it in a quite bad way by just packing all my inputs in a single Tuple and using `num_inputs = 1`. I would really appreciate getting at least your fix to solve this issue merged into upstream. Maybe in a separate PR at this is not really related to the TorchScript use case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-773154955


   > I'm curious how it integrates with PyTorch frontend. Do we convert every op not supported to relay.torchop, run BYOC flow to get TorchScript subgraphs, and send them to libtorch? Sounds interesting!
   
   This is how I'd like it to work out. I've been thinking what the best "level" is and while the operator level might seem attractive, but I I'm not sure there is an easy way to run individual operators.
   It won't help with out favourite inplace problems, though.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1061504189


   Are you on the tvm discord or so to quickly discuss?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi commented on a change in pull request #7401: RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#discussion_r822564072



##########
File path: src/runtime/contrib/libtorch/libtorch_runtime.cc
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/runtime/contrib/libtorch/libtorch_runtime.cc
+ * \brief runtime implementation for LibTorch/TorchScript.
+ */
+
+// we do not want clang to reorder our includes
+// clang-format off
+#include <tvm/runtime/module.h>
+#include <tvm/runtime/ndarray.h>
+#include <tvm/runtime/registry.h>
+
+#include <ATen/dlpack.h>
+#include <ATen/DLConvertor.h>
+#include <torch/csrc/jit/serialization/import.h>
+#include <torch/torch.h>
+
+// clang-format on
+
+#include <cstddef>
+#include <string>
+#include <tuple>
+#include <type_traits>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+namespace tvm {
+namespace runtime {
+namespace contrib {
+
+static void monly_deleter(DLManagedTensor* self) { delete self; }
+
+void run_torch_module(torch::jit::Module* module, TVMArgs args, TVMRetValue* rv) {
+  std::vector<torch::jit::IValue> inputs;
+  std::vector<torch::Tensor> outputs;
+  auto m = module->get_method("forward");
+  for (int i = 0; i < args.size(); i++) {
+    const DLTensor* arg;
+    if (args[i].IsObjectRef<NDArray>()) {
+      NDArray arr = args[i];
+      arg = arr.operator->();
+    } else {
+      arg = args[i].operator DLTensor*();
+    }
+    DLManagedTensor* inp = new DLManagedTensor{};
+    inp->dl_tensor = *arg;
+    inp->deleter = &monly_deleter;
+    // m.num_inputs includes the self argument of forward(self, ...)
+    // num_inputs - 1 is the number of (Tensor) inputs
+    if (i < static_cast<int>(m.num_inputs()) - 1) {
+      inputs.emplace_back(at::fromDLPack(inp));
+    } else {
+      outputs.emplace_back(at::fromDLPack(inp));
+    }
+  }
+  ICHECK(outputs.size() == 1) << "wrong number of args, can handle only one output";
+  torch::Tensor res = module->forward(inputs).toTensor();
+  outputs[0].copy_(res);  // too bad

Review comment:
       yeah...




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-772591355


   This is an interesting use case of byoc, cc @zhiics @comaniac 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-774255030


   Yeah, the general idea is to use this as the fallback. I can add the fallback generation here in the PR if that is better.
   Also I added a bit of a pro-con discussion regarding single op vs. program on the forum thread, if you have opinions, I'd be very grateful if you could chime in.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-919993664


   I was going to suggest using `MergeCompilerRegions`, but I saw you already using it. So I like your current approach. Sending many small functions to torch sounds like a non-trivial overhead, and I think " piece things back together into a graph" is essentially what `MergeCompilerRegions` does at Relay level.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1018688381


   M. Ruberry of the PyTorch team re-landed the update of the dlpack.h in PyTorch. If this still holds next week, it'll be exciting to bring this up to date. :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1008443597


   @t-vi Is this still WIP? I'm happy to merge whenever you think it is ready.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1008592631


   > I wonder if we could work around it by providing a "dlpack-compat" header
   
   Does this mean the same thing as https://github.com/pytorch/pytorch/pull/65047#issuecomment-972734912? (which sounds good to me). Anyway it seems we cannot count on the PyTorch-side to change, so I'd say anything that can unblock you from our side should be a good plan. cc @tqchen @junrushao1994 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-972745099


   Just a quick note that when I tried to revive this back in the summer it got a bit stalled around pytorch/pytorch#65047 .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1010432794


   Another interesting use case this fallback could enable is mmdetection https://github.com/open-mmlab/mmdetection. It has a lot of cool detection models but rely on many custom ops that cannot convert to relay.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on a change in pull request #7401: RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on a change in pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#discussion_r822551406



##########
File path: src/runtime/contrib/libtorch/libtorch_runtime.cc
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/runtime/contrib/libtorch/libtorch_runtime.cc
+ * \brief runtime implementation for LibTorch/TorchScript.
+ */
+
+// we do not want clang to reorder our includes
+// clang-format off
+#include <tvm/runtime/module.h>
+#include <tvm/runtime/ndarray.h>
+#include <tvm/runtime/registry.h>
+
+#include <ATen/dlpack.h>
+#include <ATen/DLConvertor.h>
+#include <torch/csrc/jit/serialization/import.h>
+#include <torch/torch.h>
+
+// clang-format on
+
+#include <cstddef>
+#include <string>
+#include <tuple>
+#include <type_traits>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+namespace tvm {
+namespace runtime {
+namespace contrib {
+
+static void monly_deleter(DLManagedTensor* self) { delete self; }
+
+void run_torch_module(torch::jit::Module* module, TVMArgs args, TVMRetValue* rv) {
+  std::vector<torch::jit::IValue> inputs;
+  std::vector<torch::Tensor> outputs;
+  auto m = module->get_method("forward");
+  for (int i = 0; i < args.size(); i++) {
+    const DLTensor* arg;
+    if (args[i].IsObjectRef<NDArray>()) {
+      NDArray arr = args[i];
+      arg = arr.operator->();
+    } else {
+      arg = args[i].operator DLTensor*();
+    }
+    DLManagedTensor* inp = new DLManagedTensor{};
+    inp->dl_tensor = *arg;
+    inp->deleter = &monly_deleter;
+    // m.num_inputs includes the self argument of forward(self, ...)
+    // num_inputs - 1 is the number of (Tensor) inputs
+    if (i < static_cast<int>(m.num_inputs()) - 1) {
+      inputs.emplace_back(at::fromDLPack(inp));
+    } else {
+      outputs.emplace_back(at::fromDLPack(inp));
+    }
+  }
+  ICHECK(outputs.size() == 1) << "wrong number of args, can handle only one output";
+  torch::Tensor res = module->forward(inputs).toTensor();
+  outputs[0].copy_(res);  // too bad
+  // what to do about rv?
+  // NDArray res_array = NDArray::FromDLPack(at::toDLPack(res));
+  // *rv = res_array;

Review comment:
       He yeah, with the zero copy below. :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-851457259


   > I would really appreciate getting at least your fix to solve this issue merged into upstream. Maybe in a separate PR at this is not really related to the TorchScript use case.
   
   I'm all for it, but I wouldn't know how to add tests in lieu of something using it. If you or @masahi has any opinion on this, I'd be glad to split it out...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-772828442


   I'm curious how it integrates with PyTorch frontend. Do we convert every op not supported to `relay.torchop`, run BYOC flow to get TorchScript subgraphs, and send them to libtorch? Sounds interesting!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1063277207


   Thank you @masahi !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1008583242


   @masahi So I had hoped to get the dlpack header version in PyTorch bumped (see the linked bug) but Facebook has internal uses that let it insist on the old one.
   I wonder if we could work around it by providing a "dlpack-compat" header that defines the names for the types / fields? Or I could try to somehow separate the two.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] comaniac commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
comaniac commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-773534036


   I have the same question as masahi. IIUC, after this PR, the PyTorch frontend has the capablility to convert all unsupported ops to `torchop` so that we can guarantee the flow would work. This is an interesting idea and this would be the first BYOC use case that could potentially incopreate two BYOC backends (Torch and others like TensorRT).
   
   cc @zhiics 
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] comaniac commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
comaniac commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-773534036


   I have the same question as masahi. IIUC, after this PR, the PyTorch frontend has the capablility to convert all unsupported ops to `torchop` so that we can guarantee the flow would work. This is an interesting idea and this would be the first BYOC use case that could potentially incopreate two BYOC backends (Torch and others like TensorRT).
   
   cc @zhiics 
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1008961724


   So I thought, I could wait it out, but I'll look into working around the version discrepancy in the next few weeks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on a change in pull request #7401: RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on a change in pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#discussion_r822544365



##########
File path: src/relay/backend/contrib/libtorch/libtorch_codegen.cc
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/backend/contrib/libtorch/codegen.cc
+ * \brief Implementation of libtorch codegen.
+ */
+
+#include <ATen/DLConvertor.h>
+#include <dlpack/dlpack.h>
+#include <torch/csrc/jit/api/compilation_unit.h>
+#include <torch/csrc/jit/serialization/import.h>
+#include <torch/torch.h>
+#include <tvm/relay/attrs/nn.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/op.h>
+#include <tvm/relay/transform.h>
+#include <tvm/relay/type.h>
+#include <tvm/runtime/module.h>
+#include <tvm/runtime/registry.h>
+#include <tvm/tir/op.h>
+
+#include <fstream>
+#include <numeric>
+#include <sstream>
+
+#include "../../utils.h"
+
+namespace tvm {
+namespace relay {
+namespace contrib {
+
+using namespace backend;
+
+/*! \brief Attributes of a TorchFunction node */
+struct TorchFunctionAttrs : public tvm::AttrsNode<TorchFunctionAttrs> {
+  std::string serialized_function;
+  int64_t len;
+
+  TVM_DECLARE_ATTRS(TorchFunctionAttrs, "relay.attrs.TorchFunctionAttrs") {
+    TVM_ATTR_FIELD(serialized_function).set_default("").describe("Function from fn.save(...)");
+    TVM_ATTR_FIELD(len).set_default(-1).describe("Function from fn.save(...)");
+  }
+};
+
+TVM_REGISTER_NODE_TYPE(TorchFunctionAttrs);
+
+bool TorchOpRel(const Array<Type>& types, int num_inputs, const Attrs& attrs,
+                const TypeReporter& reporter) {
+  const auto* sfattrs = attrs.as<TorchFunctionAttrs>();
+  std::stringstream str(sfattrs->serialized_function);
+  torch::jit::Module mod = torch::jit::load(str);
+
+  std::vector<torch::jit::IValue> inputs;
+  for (int i = 0; i < num_inputs; i++) {
+    auto* ty = types[i].as<TensorTypeNode>();
+    ICHECK(ty) << "only accept tensors as inputs";
+    std::vector<int64_t> shape;
+    for (const auto& s : ty->shape) {
+      auto* si = s.as<IntImmNode>();
+      if (!si) {
+        return false;
+      }
+      shape.push_back(si->value);
+    }
+    auto torchScalarType = at::toScalarType(ty->dtype);
+
+    inputs.emplace_back(torch::zeros(shape, at::TensorOptions().dtype(torchScalarType)));
+  }
+  auto res = mod.forward(inputs);
+  auto res_t = res.toTensor();
+  ICHECK((int)types.size() == num_inputs + 1) << "only single output supported";
+  Array<PrimExpr> res_sizes;
+  for (int d = 0; d < res_t.dim(); d++) {
+    res_sizes.push_back(IntImm(DataType::Int(32), res_t.size(d)));
+  }
+  reporter->Assign(types[num_inputs], TensorType(res_sizes, DataType(at::getDLDataType(res_t))));
+  return true;
+}
+
+RELAY_REGISTER_OP("torch_op")
+    .set_support_level(99)
+    .add_type_rel("TorchOpRel", TorchOpRel)
+    .set_attrs_type<TorchFunctionAttrs>();
+
+Expr MakeTorchOp(Array<Expr> args, const std::string& serialized_function) {
+  static const Op& op = Op::Get("torch_op");
+  auto attrs = make_object<TorchFunctionAttrs>();
+  attrs->serialized_function = serialized_function;
+  attrs->len = serialized_function.size();
+  return Call(op, args, Attrs(attrs), {});
+}
+
+TVM_REGISTER_GLOBAL("relay.op._make.torchop").set_body_typed(MakeTorchOp);
+
+/*!
+ * \brief The external compiler/codegen tool. It takes a Relay expression/module and
+ * compile it into a runtime module.
+ */
+runtime::Module TorchCompiler(const ObjectRef& ref) {
+  ICHECK(ref->IsInstance<FunctionNode>()) << "The input ref is expected to be a Relay function.";
+  Function func = Downcast<Function>(ref);
+  std::string func_name = backend::GetExtSymbol(func);
+
+  ICHECK(func.defined()) << "Input error: expect a Relay function.";
+  const auto* call = func->body.as<CallNode>();
+  ICHECK(call) << "Expected call node\n";
+  const auto* op_node = call->op.as<OpNode>();
+  ICHECK(op_node) << "Expect OpNode, but got " << call->op->GetTypeKey();
+  const auto op_name = GetRef<Op>(op_node)->name;
+  ICHECK(op_name == "torch_op") << "Unsupported op: " << AsText(call->op, false) << "\n";
+
+  const auto* attrs = call->attrs.as<TorchFunctionAttrs>();
+
+  // TensorRTJSONSerializer serializer(func_name, func);
+  // serializer.serialize();
+  // std::string graph_json = serializer.GetJSON();
+
+  const auto* pf = runtime::Registry::Get("runtime.torch_runtime_create");
+  ICHECK(pf != nullptr) << "Cannot find Torch runtime module create function.";

Review comment:
       Thank you! I was unsure whether runtime and codegen cross-links should go through the registry, I'll fix that.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi merged pull request #7401: RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi merged pull request #7401:
URL: https://github.com/apache/tvm/pull/7401


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on a change in pull request #7401: RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on a change in pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#discussion_r822551406



##########
File path: src/runtime/contrib/libtorch/libtorch_runtime.cc
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/runtime/contrib/libtorch/libtorch_runtime.cc
+ * \brief runtime implementation for LibTorch/TorchScript.
+ */
+
+// we do not want clang to reorder our includes
+// clang-format off
+#include <tvm/runtime/module.h>
+#include <tvm/runtime/ndarray.h>
+#include <tvm/runtime/registry.h>
+
+#include <ATen/dlpack.h>
+#include <ATen/DLConvertor.h>
+#include <torch/csrc/jit/serialization/import.h>
+#include <torch/torch.h>
+
+// clang-format on
+
+#include <cstddef>
+#include <string>
+#include <tuple>
+#include <type_traits>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+namespace tvm {
+namespace runtime {
+namespace contrib {
+
+static void monly_deleter(DLManagedTensor* self) { delete self; }
+
+void run_torch_module(torch::jit::Module* module, TVMArgs args, TVMRetValue* rv) {
+  std::vector<torch::jit::IValue> inputs;
+  std::vector<torch::Tensor> outputs;
+  auto m = module->get_method("forward");
+  for (int i = 0; i < args.size(); i++) {
+    const DLTensor* arg;
+    if (args[i].IsObjectRef<NDArray>()) {
+      NDArray arr = args[i];
+      arg = arr.operator->();
+    } else {
+      arg = args[i].operator DLTensor*();
+    }
+    DLManagedTensor* inp = new DLManagedTensor{};
+    inp->dl_tensor = *arg;
+    inp->deleter = &monly_deleter;
+    // m.num_inputs includes the self argument of forward(self, ...)
+    // num_inputs - 1 is the number of (Tensor) inputs
+    if (i < static_cast<int>(m.num_inputs()) - 1) {
+      inputs.emplace_back(at::fromDLPack(inp));
+    } else {
+      outputs.emplace_back(at::fromDLPack(inp));
+    }
+  }
+  ICHECK(outputs.size() == 1) << "wrong number of args, can handle only one output";
+  torch::Tensor res = module->forward(inputs).toTensor();
+  outputs[0].copy_(res);  // too bad
+  // what to do about rv?
+  // NDArray res_array = NDArray::FromDLPack(at::toDLPack(res));
+  // *rv = res_array;

Review comment:
       I had hoped that one might be able to avoid the copy, but apparently not easily.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1008597919


   As commented by the author of PyTorchTVM, https://github.com/apache/tvm-rfcs/pull/25#issuecomment-908041324, many people are interested in this feature. Also people are actively talking about deeper integration of TVM into PyTorch workflow. So we should definitely land this.
    
   As for me personally, I want to try out this feature to import models having fft-related ops (specifically, this cool model https://github.com/raoyongming/GFNet). FFT ops won't come to TVM any time soon. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi commented on a change in pull request #7401: RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi commented on a change in pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#discussion_r821992527



##########
File path: src/relay/backend/contrib/libtorch/libtorch_codegen.cc
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/relay/backend/contrib/libtorch/codegen.cc
+ * \brief Implementation of libtorch codegen.
+ */
+
+#include <ATen/DLConvertor.h>
+#include <dlpack/dlpack.h>
+#include <torch/csrc/jit/api/compilation_unit.h>
+#include <torch/csrc/jit/serialization/import.h>
+#include <torch/torch.h>
+#include <tvm/relay/attrs/nn.h>
+#include <tvm/relay/expr_functor.h>
+#include <tvm/relay/op.h>
+#include <tvm/relay/transform.h>
+#include <tvm/relay/type.h>
+#include <tvm/runtime/module.h>
+#include <tvm/runtime/registry.h>
+#include <tvm/tir/op.h>
+
+#include <fstream>
+#include <numeric>
+#include <sstream>
+
+#include "../../utils.h"
+
+namespace tvm {
+namespace relay {
+namespace contrib {
+
+using namespace backend;
+
+/*! \brief Attributes of a TorchFunction node */
+struct TorchFunctionAttrs : public tvm::AttrsNode<TorchFunctionAttrs> {
+  std::string serialized_function;
+  int64_t len;
+
+  TVM_DECLARE_ATTRS(TorchFunctionAttrs, "relay.attrs.TorchFunctionAttrs") {
+    TVM_ATTR_FIELD(serialized_function).set_default("").describe("Function from fn.save(...)");
+    TVM_ATTR_FIELD(len).set_default(-1).describe("Function from fn.save(...)");
+  }
+};
+
+TVM_REGISTER_NODE_TYPE(TorchFunctionAttrs);
+
+bool TorchOpRel(const Array<Type>& types, int num_inputs, const Attrs& attrs,
+                const TypeReporter& reporter) {
+  const auto* sfattrs = attrs.as<TorchFunctionAttrs>();
+  std::stringstream str(sfattrs->serialized_function);
+  torch::jit::Module mod = torch::jit::load(str);
+
+  std::vector<torch::jit::IValue> inputs;
+  for (int i = 0; i < num_inputs; i++) {
+    auto* ty = types[i].as<TensorTypeNode>();
+    ICHECK(ty) << "only accept tensors as inputs";
+    std::vector<int64_t> shape;
+    for (const auto& s : ty->shape) {
+      auto* si = s.as<IntImmNode>();
+      if (!si) {
+        return false;
+      }
+      shape.push_back(si->value);
+    }
+    auto torchScalarType = at::toScalarType(ty->dtype);
+
+    inputs.emplace_back(torch::zeros(shape, at::TensorOptions().dtype(torchScalarType)));
+  }
+  auto res = mod.forward(inputs);
+  auto res_t = res.toTensor();
+  ICHECK((int)types.size() == num_inputs + 1) << "only single output supported";
+  Array<PrimExpr> res_sizes;
+  for (int d = 0; d < res_t.dim(); d++) {
+    res_sizes.push_back(IntImm(DataType::Int(32), res_t.size(d)));
+  }
+  reporter->Assign(types[num_inputs], TensorType(res_sizes, DataType(at::getDLDataType(res_t))));
+  return true;
+}
+
+RELAY_REGISTER_OP("torch_op")
+    .set_support_level(99)
+    .add_type_rel("TorchOpRel", TorchOpRel)
+    .set_attrs_type<TorchFunctionAttrs>();
+
+Expr MakeTorchOp(Array<Expr> args, const std::string& serialized_function) {
+  static const Op& op = Op::Get("torch_op");
+  auto attrs = make_object<TorchFunctionAttrs>();
+  attrs->serialized_function = serialized_function;
+  attrs->len = serialized_function.size();
+  return Call(op, args, Attrs(attrs), {});
+}
+
+TVM_REGISTER_GLOBAL("relay.op._make.torchop").set_body_typed(MakeTorchOp);
+
+/*!
+ * \brief The external compiler/codegen tool. It takes a Relay expression/module and
+ * compile it into a runtime module.
+ */
+runtime::Module TorchCompiler(const ObjectRef& ref) {
+  ICHECK(ref->IsInstance<FunctionNode>()) << "The input ref is expected to be a Relay function.";
+  Function func = Downcast<Function>(ref);
+  std::string func_name = backend::GetExtSymbol(func);
+
+  ICHECK(func.defined()) << "Input error: expect a Relay function.";
+  const auto* call = func->body.as<CallNode>();
+  ICHECK(call) << "Expected call node\n";
+  const auto* op_node = call->op.as<OpNode>();
+  ICHECK(op_node) << "Expect OpNode, but got " << call->op->GetTypeKey();
+  const auto op_name = GetRef<Op>(op_node)->name;
+  ICHECK(op_name == "torch_op") << "Unsupported op: " << AsText(call->op, false) << "\n";
+
+  const auto* attrs = call->attrs.as<TorchFunctionAttrs>();
+
+  // TensorRTJSONSerializer serializer(func_name, func);
+  // serializer.serialize();
+  // std::string graph_json = serializer.GetJSON();
+
+  const auto* pf = runtime::Registry::Get("runtime.torch_runtime_create");
+  ICHECK(pf != nullptr) << "Cannot find Torch runtime module create function.";

Review comment:
       For cpp code, it's better to expose this function in a header and directly call it.

##########
File path: src/runtime/contrib/libtorch/libtorch_runtime.cc
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/runtime/contrib/libtorch/libtorch_runtime.cc
+ * \brief runtime implementation for LibTorch/TorchScript.
+ */
+
+// we do not want clang to reorder our includes
+// clang-format off
+#include <tvm/runtime/module.h>
+#include <tvm/runtime/ndarray.h>
+#include <tvm/runtime/registry.h>
+
+#include <ATen/dlpack.h>
+#include <ATen/DLConvertor.h>
+#include <torch/csrc/jit/serialization/import.h>
+#include <torch/torch.h>
+
+// clang-format on
+
+#include <cstddef>
+#include <string>
+#include <tuple>
+#include <type_traits>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+namespace tvm {
+namespace runtime {
+namespace contrib {
+
+static void monly_deleter(DLManagedTensor* self) { delete self; }
+
+void run_torch_module(torch::jit::Module* module, TVMArgs args, TVMRetValue* rv) {
+  std::vector<torch::jit::IValue> inputs;
+  std::vector<torch::Tensor> outputs;
+  auto m = module->get_method("forward");
+  for (int i = 0; i < args.size(); i++) {
+    const DLTensor* arg;
+    if (args[i].IsObjectRef<NDArray>()) {
+      NDArray arr = args[i];
+      arg = arr.operator->();
+    } else {
+      arg = args[i].operator DLTensor*();
+    }
+    DLManagedTensor* inp = new DLManagedTensor{};
+    inp->dl_tensor = *arg;
+    inp->deleter = &monly_deleter;
+    // m.num_inputs includes the self argument of forward(self, ...)
+    // num_inputs - 1 is the number of (Tensor) inputs
+    if (i < static_cast<int>(m.num_inputs()) - 1) {
+      inputs.emplace_back(at::fromDLPack(inp));
+    } else {
+      outputs.emplace_back(at::fromDLPack(inp));
+    }
+  }
+  ICHECK(outputs.size() == 1) << "wrong number of args, can handle only one output";
+  torch::Tensor res = module->forward(inputs).toTensor();
+  outputs[0].copy_(res);  // too bad

Review comment:
       There is `SetOutputZeroCopy`, can we use that here?
   
   https://github.com/apache/tvm/blob/f9d8c2b99615f074fd7b0ae95d04825ab443fa33/src/runtime/graph_executor/graph_executor.cc#L196

##########
File path: src/runtime/contrib/libtorch/libtorch_runtime.cc
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file src/runtime/contrib/libtorch/libtorch_runtime.cc
+ * \brief runtime implementation for LibTorch/TorchScript.
+ */
+
+// we do not want clang to reorder our includes
+// clang-format off
+#include <tvm/runtime/module.h>
+#include <tvm/runtime/ndarray.h>
+#include <tvm/runtime/registry.h>
+
+#include <ATen/dlpack.h>
+#include <ATen/DLConvertor.h>
+#include <torch/csrc/jit/serialization/import.h>
+#include <torch/torch.h>
+
+// clang-format on
+
+#include <cstddef>
+#include <string>
+#include <tuple>
+#include <type_traits>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+namespace tvm {
+namespace runtime {
+namespace contrib {
+
+static void monly_deleter(DLManagedTensor* self) { delete self; }
+
+void run_torch_module(torch::jit::Module* module, TVMArgs args, TVMRetValue* rv) {
+  std::vector<torch::jit::IValue> inputs;
+  std::vector<torch::Tensor> outputs;
+  auto m = module->get_method("forward");
+  for (int i = 0; i < args.size(); i++) {
+    const DLTensor* arg;
+    if (args[i].IsObjectRef<NDArray>()) {
+      NDArray arr = args[i];
+      arg = arr.operator->();
+    } else {
+      arg = args[i].operator DLTensor*();
+    }
+    DLManagedTensor* inp = new DLManagedTensor{};
+    inp->dl_tensor = *arg;
+    inp->deleter = &monly_deleter;
+    // m.num_inputs includes the self argument of forward(self, ...)
+    // num_inputs - 1 is the number of (Tensor) inputs
+    if (i < static_cast<int>(m.num_inputs()) - 1) {
+      inputs.emplace_back(at::fromDLPack(inp));
+    } else {
+      outputs.emplace_back(at::fromDLPack(inp));
+    }
+  }
+  ICHECK(outputs.size() == 1) << "wrong number of args, can handle only one output";
+  torch::Tensor res = module->forward(inputs).toTensor();
+  outputs[0].copy_(res);  // too bad
+  // what to do about rv?
+  // NDArray res_array = NDArray::FromDLPack(at::toDLPack(res));
+  // *rv = res_array;

Review comment:
       Remove these since `outputs[0].copy_(res);` does the job of sending the output to TVM?
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] t-vi commented on pull request #7401: WIP/RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
t-vi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1061478284


   Hi, so I rebased this finally and it all compiles and runs one test against a current PyTorch master, so I think I'm back in business with this PR (unless it has been obsoleted, but from what I understand, the bridge is in the other direction).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [tvm] masahi commented on pull request #7401: RFC: initial stab at TorchScript fallback

Posted by GitBox <gi...@apache.org>.
masahi commented on pull request #7401:
URL: https://github.com/apache/tvm/pull/7401#issuecomment-1061554624


   yeah I've just logged in.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org