You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "li-liwen (via GitHub)" <gi...@apache.org> on 2025/01/01 15:04:53 UTC
[D] GPU passthrough support in KVM [cloudstack]
GitHub user li-liwen created a discussion: GPU passthrough support in KVM
Hi all. I've been trying to setup a proper GPU enabled KVM cluster for weeks. I cannot find any official document or complete tutorial on this topic. It seems that GPU is working on Xen, but I'm a fan of Ceph and KVM, so I really would like to make it work on KVM.
Currently, I could only find some recent discussion indicating it's not working on KVM: https://github.com/apache/cloudstack/discussions/10112#discussion-7693845
This issue has been resolved, but the writer indicates it's only working in Xen, and I can't get it working even in 4.20. #9573
There's also a tutorial which not accessible any more on how to use extra config to pass GPU to one instance.
https://lab.piszki.pl/cloudstack-kvm-and-running-vm-with-vgpu/
I'm looking for a solution that can create a compute offering with GPU support, so that users can create instances with GPU passed through without the help of administrator (or other type of API). I'm wondering if it's possible to do that with KVM, and other non-datacenter GPUs like RTX 4090. If it can be done I'm happy to update the document for that.
Thanks for answering.
GitHub link: https://github.com/apache/cloudstack/discussions/10155
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] GPU passthrough support in KVM [cloudstack]
Posted by "vishesh92 (via GitHub)" <gi...@apache.org>.
GitHub user vishesh92 added a comment to the discussion: GPU passthrough support in KVM
@li-liwen Are you doing GPU passthrough using the extra config or are you using libvirt hooks?
GitHub link: https://github.com/apache/cloudstack/discussions/10155#discussioncomment-11759544
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] GPU passthrough support in KVM [cloudstack]
Posted by "rajujith (via GitHub)" <gi...@apache.org>.
GitHub user rajujith added a comment to the discussion: GPU passthrough support in KVM
@li-liwen can you try the workarounds mentioned here: https://github.com/apache/cloudstack/discussions/10076#discussioncomment-11540274
GitHub link: https://github.com/apache/cloudstack/discussions/10155#discussioncomment-11722691
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] GPU passthrough support in KVM [cloudstack]
Posted by "li-liwen (via GitHub)" <gi...@apache.org>.
GitHub user li-liwen added a comment to the discussion: GPU passthrough support in KVM
@vishesh92 I'm using Libvirt hooks. Specifically, the groovy agent hooks introduced in #3839. I'm trying to write a script that can dynamically check available GPUs (by maintaining an extra json file containing all GPUs). I'm still working on this, but I think it should work. However, when there's no available GPU, the instance will either boot normally without GPU (throw an exception in the groovy script), or enter error state (use sys.exit(1) instead). Neither is ideal. But I cannot think of a better way to do that.
GitHub link: https://github.com/apache/cloudstack/discussions/10155#discussioncomment-11760036
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] GPU passthrough support in KVM [cloudstack]
Posted by "li-liwen (via GitHub)" <gi...@apache.org>.
GitHub user li-liwen closed a discussion: GPU passthrough support in KVM
Hi all. I've been trying to setup a proper GPU enabled KVM cluster for weeks. I cannot find any official document or complete tutorial on this topic. It seems that GPU is working on Xen, but I'm a fan of Ceph and KVM, so I really would like to make it work on KVM.
Currently, I could only find some recent discussion indicating it's not working on KVM: https://github.com/apache/cloudstack/discussions/10112#discussion-7693845
This issue has been resolved, but the writer indicates it's only working in Xen, and I can't get it working even in 4.20. #9573
There's also a tutorial which not accessible any more on how to use extra config to pass GPU to one instance.
https://lab.piszki.pl/cloudstack-kvm-and-running-vm-with-vgpu/
I'm looking for a solution that can create a compute offering with GPU support, so that users can create instances with GPU passed through without the help of administrator (or other type of API). I'm wondering if it's possible to do that with KVM, and other non-datacenter GPUs like RTX 4090. If it can be done I'm happy to update the document for that.
Thanks for answering.
GitHub link: https://github.com/apache/cloudstack/discussions/10155
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] GPU passthrough support in KVM [cloudstack]
Posted by "rajujith (via GitHub)" <gi...@apache.org>.
GitHub user rajujith added a comment to the discussion: GPU passthrough support in KVM
@li-liwen this talk about the GPU feature proposal should have the answer for you, the short answer is it will need some development. https://www.youtube.com/watch?v=UyJTcy69ncg
GitHub link: https://github.com/apache/cloudstack/discussions/10155#discussioncomment-12161489
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] GPU passthrough support in KVM [cloudstack]
Posted by "li-liwen (via GitHub)" <gi...@apache.org>.
GitHub user li-liwen added a comment to the discussion: GPU passthrough support in KVM
@rajujith Thanks for answering! I have tried the method and got GPU Passthrough working. Cheers!
However, I do need some sort of orchestration. For example, I have four 4090 GPUs on one machine, and I would like to provide different compute offerings with one, two or all four GPUs. If a user tries to deploy a new instance (or scale from the old one) when not enough GPUs are available, the hook script will fail and the instance will go to Error state, which requires administrator operation. The desired behavior is to let CloudStack know how many GPU available in some way, and prevent trying to deploy instance that cannot fit.
Is there any recommended way to do this? Should I let the Groovy hook script interact with CloudStack management sql database directly? Or should I modify the GPU offering in the CloudStack UI somehow?
Many thanks for helping me and to every contributor of this project.
GitHub link: https://github.com/apache/cloudstack/discussions/10155#discussioncomment-11726204
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] GPU passthrough support in KVM [cloudstack]
Posted by "li-liwen (via GitHub)" <gi...@apache.org>.
GitHub user li-liwen added a comment to the discussion: GPU passthrough support in KVM
Thank you all for answering. I've used the libvirt hooks as a work around for GPU passthrough. However, if there's not enough GPU, the VM will enter error state. Is there any way for user or admin to modify the service offering of the error VM, so that it can be reattempted to deploy?
GitHub link: https://github.com/apache/cloudstack/discussions/10155#discussioncomment-12113899
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org