You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/03/03 03:46:13 UTC
[GitHub] [tvm] zhuochenKIDD opened a new pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
zhuochenKIDD opened a new pull request #7573:
URL: https://github.com/apache/tvm/pull/7573
We are currently using graph runtime to run some CTR models on NV-GPU, for our in-house model (around 100 nodes in tvm json graph ) cuGraphLaunch can reduce 5% to 10% percent latency vs the original for-loop cuda kernel launch.
So I wonder if the extension might benefits other workloads, I haven't test other types of models.
This is a POC, will supplement demos/docs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] FrozenGene commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-792546803
@zhuochenKIDD Besides requires CUDA 10, does cuda graph require special version of GPU hardware? like Tesla or Ampere?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tqchen commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-791477306
@zhuochenKIDD please checkout the lint error from the CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
comaniac commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-791572612
In terms of the interface, can we use `GraphRuntimeFactory` to dispatch the extended runtime so that it would be more straightforward for users?
cc @FrozenGene @zhiics @icemelon9 @vinx13
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] FrozenGene commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-793327449
> > If i don't understand it wrong, graph runtime we have done it before, but have not done for VM or other runtime.
>
> This PR is for graph runtime so it's not an issue for other runtimes? My concern is if we can reduce the API complexity from users. Now users need to use `tvm.contrib.cu_graph.cugraph_runtime.create(...)` to make use of this runtime, but this is not recommeneded. We should make all graph runtimes craeted from `graph_runtime.GraphModule`.
Not it. I just meant our `GraphRuntimeFactory` has designed for this purpose you mentioned before. However, `GraphRuntimeFactory` is applied only for `GraphRuntime` now, not applied for VM or other runtime. But `GraphRuntimeFactory` could be used for the purpose you mentioned.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zhuochenKIDD commented on a change in pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
zhuochenKIDD commented on a change in pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#discussion_r587188515
##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -481,6 +481,15 @@ TVM_DLL int TVMStreamFree(int device_type, int device_id, TVMStreamHandle stream
*/
TVM_DLL int TVMSetStream(int device_type, int device_id, TVMStreamHandle handle);
+
+TVM_DLL int TVMStreamBeginCapture(int device_type, int device_id, TVMStreamHandle stream);
Review comment:
thx, already removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] FrozenGene commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-792666127
> > @zhuochenKIDD Do you have a guess as to why this is faster than the for-loop launch approach?
>
> @tkonolige not sure why it's faster，it's based on test，and depends on workloads.
> I guest for my model, it has many tiny kernels which is more kernel-launch bound, and cuda-graph might reduce kernel-launch overhead, I will do more profiling & analysis，do you have some suggestion?
I think the answer is here: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs
> This allows a graph to be defined once and then launched repeatedly. Separating out the definition of a graph from its execution enables a number of optimizations: first, CPU launch costs are reduced compared to streams, because much of the setup is done in advance; second, presenting the whole workflow to CUDA enables optimizations which might not be possible with the piecewise work submission mechanism of streams.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] FrozenGene commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-792546254
> In terms of the interface, can we use `GraphRuntimeFactory` to dispatch the extended runtime so that it would be more straightforward for users?
>
> cc @FrozenGene @zhiics @icemelon9 @vinx13
If i don't understand it wrong, graph runtime we have done it before, but have not done for VM or other runtime.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zhuochenKIDD closed pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
zhuochenKIDD closed pull request #7573:
URL: https://github.com/apache/tvm/pull/7573
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zhuochenKIDD commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
zhuochenKIDD commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-791124864
> @zhuochenKIDD Do you have a guess as to why this is faster than the for-loop launch approach?
@tkonolige not sure why it's faster，it's based on test，and depends on workloads.
I guest for my model, it has many tiny kernels which is more kernel-launch bound, and cuda-graph might reduce kernel-launch overhead, I will do more profiling & analysis，do you have some suggestion?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tkonolige commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
tkonolige commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-790877164
@zhuochenKIDD Do you have a guess as to why this is faster than the for-loop launch approach?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] comaniac commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
comaniac commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-792951538
> If i don't understand it wrong, graph runtime we have done it before, but have not done for VM or other runtime.
This PR is for graph runtime so it's not an issue for other runtimes? My concern is if we can reduce the API complexity from users. Now users need to use `tvm.contrib.cu_graph.cugraph_runtime.create(...)` to make use of this runtime, but this is not recommeneded. We should make all graph runtimes craeted from `graph_runtime.GraphModule`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] FrozenGene commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-792666939
> @zhuochenKIDD Besides requires CUDA 10, does cuda graph require special version of GPU hardware? like Tesla or Ampere?
I find the answer: https://docs.nvidia.com/cuda/cuda-samples/index.html#simple-cuda-graphs no special version of GPU hardware required. It is optimization at software level.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tqchen commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-789833446
cc @trevor-m @antinucleon @merrymercy please help to review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tqchen commented on a change in pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#discussion_r586551771
##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -481,6 +481,15 @@ TVM_DLL int TVMStreamFree(int device_type, int device_id, TVMStreamHandle stream
*/
TVM_DLL int TVMSetStream(int device_type, int device_id, TVMStreamHandle handle);
+
+TVM_DLL int TVMStreamBeginCapture(int device_type, int device_id, TVMStreamHandle stream);
Review comment:
Given cuGraph is specific to CUDA atm, let us not introduce it to the DeviceAPI for now, instead just use the cuda api in cugraph runtime
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zhuochenKIDD commented on pull request #7573: [Runtime] Extend Graph Runtime To Support Cuda Graph Launch
Posted by GitBox <gi...@apache.org>.
zhuochenKIDD commented on pull request #7573:
URL: https://github.com/apache/tvm/pull/7573#issuecomment-793890206
I opened a more clean PR, close this.
https://github.com/apache/tvm/pull/7616
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org