You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "tqchen (via GitHub)" <gi...@apache.org> on 2025/05/07 15:29:39 UTC
[PR] [FFI][FEAT] AutoDLPack for taking external tensor objects. [tvm]
tqchen opened a new pull request, #17927:
URL: https://github.com/apache/tvm/pull/17927
This PR introduces autodlpack feature to the tvm ffi. When an ffi Function takes Tensor argument that conforms to DLPack it automatically imports into NDArray and pass as argument.
The feature will allow compiled function to directly take torch.Tensor as input argument without extra set of changes.
We also added a benchmark script to measure the overall ffi overhead. One thing to note is that there is still continuguous and alignment requirement that is needed by underlying DSL compiler, as of now we use a global value. So x.continugous is still needed before passing the argument if tranpose or other ops are performed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FFI][FEAT] AutoDLPack for taking external tensor objects [tvm]
Posted by "Hzfengsy (via GitHub)" <gi...@apache.org>.
Hzfengsy merged PR #17927:
URL: https://github.com/apache/tvm/pull/17927
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FFI][FEAT] AutoDLPack for taking external tensor objects [tvm]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on PR #17927:
URL: https://github.com/apache/tvm/pull/17927#issuecomment-2859152015
## Benchmark
Env CPU: AMD Ryzen 9 7950X
```bash
> python ffi/scripts/benchmark_dlpack.py
-----------------------------
Benchmark f(x, y, z) overhead
-----------------------------
numpy.add 1.921653747558594e-07 sec/call
torch.add[cpu] 6.330013275146484e-07 sec/call
torch.add[cuda] 2.330756187438965e-06 sec/call
tvm.ffi.nop 3.983736038208008e-07 sec/call
tvm.ffi.nop+from_dlpack(torch) 4.368019104003906e-06 sec/call
tvm.ffi.nop+from_dlpack(numpy) 1.1694192886352538e-06 sec/call
tvm.ffi.nop+from_dlpack(tvm) 1.4580249786376954e-06 sec/call
tvm.ffi.nop+from_dlpack(torch.utils) 3.2754182815551756e-06 sec/call
tvm.ffi.nop.autodlpack(torch[cpu]) 3.567361831665039e-06 sec/call
tvm.ffi.nop.autodlpack(torch[cuda]) 3.5606861114501952e-06 sec/call
tvm.ffi.nop.autodlpack(numpy) 1.6696929931640624e-06 sec/call
-------------------------------
Benchmark x.__dlpack__ overhead
-------------------------------
torch.utils.dlpack.to_dlpack 4.5762062072753906e-07 sec/call
torch.__dlpack__ 9.840965270996094e-07 sec/call
numpy.__dlpack__ 5.011558532714844e-08 sec/call
tvm.__dlpack__ 1.5852451324462892e-07 sec/call
---------------------------------------------------
Benchmark x.__dlpack__(max_version=(1,1)) overhead
---------------------------------------------------
torch.__dlpack__(max_version=(1,1)) Tensor.__dlpack__() got an unexpected keyword argument 'max_version'
numpy.__dlpack__(max_version=(1,1)) 6.172657012939454e-08 sec/call
tvm.__dlpack__(max_version=(1,1)) 1.720428466796875e-07 sec/call
## Discussions
- First, we can see that the overall FFI overhead of python c++ is roughly at 0.2us -3us level
- Notably, each torch.add eager call in cuda is around 2.4 us
- AutoDLPack as of now can get to about 3.6us for a call of f(x, y, z) that needs three import calls
- One can observe that `torch.__dlpack__` overhead is larger than `numpy.__dlpack__`
- AutoDLPack from numpy arguments have about 1.6us overhead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FFI][FEAT] AutoDLPack for taking external tensor objects [tvm]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on PR #17927:
URL: https://github.com/apache/tvm/pull/17927#issuecomment-2859232374
cc @MasterJH5574 @Hzfengsy @LeiWang1999
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org