You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "wangwei0537 (via GitHub)" <gi...@apache.org> on 2025/02/27 13:29:21 UTC
[I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
wangwei0537 opened a new issue, #10479:
URL: https://github.com/apache/cloudstack/issues/10479
### The required feature described as a wish
CloudStack's network functions and management are usually implemented through System Virtual Machines (SVMs), especially network-related functions such as routing, NAT, VPN, DHCP, etc. How can we integrate with physical network devices to replace system virtual machines? Are there any relevant configuration cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "btzq (via GitHub)" <gi...@apache.org>.
btzq commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2692728611
FYI, if you are going to test a Hypervisor Failure, to trigger the VM HA to auto start on another healthy hypervisor, i suggest to simulate a force shut down or Power Off instead. Just like how you simulate a real scenario is going to happen.
There have been reports that restarting a hypervisor doesnt trigger the VM HA cause it isnt considered an outage.
And there are some parameters u need to tune to reduce the time Cloudstack Management server needs to realise the hypervisor is down. If not, by default, it takes a loooong time and multiple people have reported this.
But on the other than, you want to make sure not to tune these parameters too low, because you might cause a ‘split brain scenario’ (depending on your underlying storage). What if the hypervisor is still online? But Cloudstack Management cant reach it and thinks its down? So it started the VM in another hypervisor causing 2 VM to exist and talk to the same storage volume? Becareful… dependin on the storage u use (NFS, Linstor etc), there will be different recommendations of avoiding split brain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "wangwei0537 (via GitHub)" <gi...@apache.org>.
wangwei0537 commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2691783465
You have understood the question I raised. Thank you for telling me so much. I used cloustack three years ago. At that time, I restarted the manager node in the test environment. At that time, there was only one manager node. The manager UI could be accessed normally, but the virtual machine responsible for the network always had problems. Although it seemed to be up, the network was not normal and could not communicate with the virtual machine. At that time, I used openvswitch to transparently pass the VLAN of the physical network device to openvswitch. It has been a while, so I will study it carefully. Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "wangwei0537 (via GitHub)" <gi...@apache.org>.
wangwei0537 commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2688129644
Routers, switches, VPN gateways, firewalls and other physical devices
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "wangwei0537 (via GitHub)" <gi...@apache.org>.
wangwei0537 commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2690271365
As you said, VPC will use system VMs such as virtual routers to provide the functions required for VPC to work. For example, routing, DHCP, load balancing, etc. I may have a problem with my expression. What I actually want to express is whether we can abandon the use of system VMs to provide the functions required by VPC.
I personally think that system VMs are not a very good design. They don’t seem to be very stable. Every time I restart the server, it always hangs up and there is no way to recover.
I think the biggest advantage of CloudStack is that it is fast enough, but it seems to be a little worse than OpenStack in the design of the network model. This is my real thought. Maybe I don’t know enough about CloudStack. Thank you for your reply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "btzq (via GitHub)" <gi...@apache.org>.
btzq commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2692725813
Mm your scenario sounds unusual.
Usually Management and Management Database is installed on its own dedicated server.
While hypervisor, virtual router etc is another physical server.
If management server is restarted (or goes offline), all Virtual Routers, Virtual Machines etc should still run as usual.
I’ve definitely tried this many times and can confirm this behavior.
But i wasnt using OpenvSwitch, im using vxlan and default networking. Might need to ask someone who is familiar with OpenVSwitch instead…
But one thing i learnt about using Cloudstack, is that if youre going to use a feature, and you cant afford time to R&D, then use only the common features that everyone else is using. Then at least, most of the times, things will work normally, and if you encounter any issue, it is easier for the community to support it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "btzq (via GitHub)" <gi...@apache.org>.
btzq commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2714181672
@dcontiveros out of curiosity, whats your use case of openvswitch?
For my side, we were looking at something like macvtap or more advanced networking so we can achieve kernel bypass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "wangwei0537 (via GitHub)" <gi...@apache.org>.
wangwei0537 commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2689266090
In OpenStack, Neutron components provide a variety of network functions. The core plugin implements internal layer 2 communication, and the L3 Agent implements layer 3 (L3) network functions to help tenants' virtual machines (VMs) communicate with the external network. Their network functions are not implemented through system virtual machines. I mean, is there a way for CloudStack to implement network functions without using system virtual machines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "btzq (via GitHub)" <gi...@apache.org>.
btzq commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2689460435
Mmm its like this:
VPCs Will use system VMs like the Virtual Router to provide the functionality for VPCs to work. Such as Routing, DHCP, Load Balancing etc
If you dont want to use Virtual Router, you can create L2 Networks using Cloudstack. Its just a simple network that doesnt do anything on its own. So, u will need to have some sort of ‘device’ to provide features such as firewall, routing etc. To do that, Cloudstack has a feature called VNF (Virtual Network Function). It is where you can boot up a VM using an ISO, that had a virtualised network applicance in it. So the most common use case of it is things like PFsense, vyos etc.
It is possible to passthrough hardware appliances to a VM which allows u to have some form of integration with ‘hardware’. But that is not recommended unless u have very specific use cases (and u have to manage it outside of cloudstack). VNF is recommended instead because u can resize the VM according to ur load and save costs. Furthemore, hardware based appliances are pretty expensive… software based can be cheaper and have same performance.
In the cloudstack docs, though it talks about some possible integration with hardwares such as netscalet and palo alto, i highly dont recommend it. It was written maybe 9 years ago and is no longer maintained.
But again it is easier to answer if u share ur specific goal? Like what hardware are u trying to integrate and why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-3688427725
This issue is stale because it has been open for 120 days with no activity. It may be removed by administrators of this project at any time. Remove the stale label or comment to request for removal of it to prevent this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2687991946
@wangwei0537
what physical network devices do you have ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "wangwei0537 (via GitHub)" <gi...@apache.org>.
wangwei0537 commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2690245518
thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "dcontiveros (via GitHub)" <gi...@apache.org>.
dcontiveros commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2733413514
I was under the impression that VDPA with Mellanox connectx6-DX would solve the live migration issue. I think we are discussing two different things. Let's realign. I was under the impression that you could use both SR-IOV and DPDK in tandem on OVS. What I wanted to do was:
1. Get userspace networking working with SR-IOV or DPDK via representor ports
2. being able to live migrate via VDPA.
All of this is still in POC I am actually labbing this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "btzq (via GitHub)" <gi...@apache.org>.
btzq commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2733397649
We tried to achieve kernell bypass ourselves using either SRIOV
The problem is that by bypassing the kernel, the traffic doesnt go thru the Virtual Router. Which we werent sure if it would break anything.
Also, SRIOV will break LACP and Live Migrations…
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "btzq (via GitHub)" <gi...@apache.org>.
btzq commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2691190996
@wangwei0537
I think what you are trying to ask, is if it is possible to create and use a Cloudstack VPC, but use other VNF or Hardware Appliances to substitute/replace the systemVMs such a Virtual Router, while still being able to retain all the native functionalities of a Cloudstack VPC?
If this is the case, the answer is no.
I understand where you come from that there are some edge cases with Cloudstack VPCs, but based on my observations in this community, most of the issues are due to configuration issues, or lack of knowledge because there is a learning curve to learning how to setup, configure and scale Cloudstack as a Cloud Operator. But at the same time, i understand if you found bugs, i have encountered my fair share too.
Cloudstack VPC may have an impression that it has a bad design because the Virtual Router is taking on many functions such as Routing, DHCP, Load Balancing, Static NAT, etc all in 1 Virtual Machine. It makes it difficult to scale, and the Virtual Router is not optimized at the moment, which means you get a performance/bandwidth issue when you scale to a certain throughput. If i remember correctly, it is like 2.5-5 Gbps (Total Throughput)? But i might remember it incorrectly.
There is also a lack of more advanced networking options. The most advanced would be the support of OpenVSwitch (which i dont see many people using at the moment, or maybe i cant see it). There isnt any more advanced networking support such as macvtap, static routes for VPCs (currently Static Routes for Private Gateway are only possible). Ive reported that ticket here, last year, and its being scheduled for 4.21 at the moment: https://github.com/apache/cloudstack/issues/9791
Bottom line, yes, i think there is lots of room to improve, and the community knows this. Which is why Cloudstack community tried to integrate Tungsten Fabric somewhere last year. I believe the first phase of integration was done, but unfortunately the Tungsten Fabric project was shut down. Some people have revived it under OpenSDN, but because the project is still shaky at the moment, the community seemed to have stop work on that.
If that had suceeded, i think you would have been able to achieve features very similar to Openstack in terms of more advanced networking options. But of course, the trade off is that you will need to have very familiar and knowledable people to operate and configure it.
However, i still think Cloudstack is easier (and cheaper) to setup than Openstack. Openstack has way too many components and microservices, and having to set it up would require a really really experienced team who knows infrastructure, networking, software development etc. To me, Openstack is a collection of microservices and multiple projects that Cloud Operators need to stitch together manually to make it work. From a cost perspective as well, it is costlier to setup (Capex) and maintain (Opex). Cloudstack, still has its use cases and can do lots of things. You just have to know what you want to achieve.
By the way, your scenario where you turned off the server, and the service hangs, dont not sounds normal.
- Are you using Hyperconverged, or Dissagregated Architecture?
- What server did you restart excatly? Management Server? Compute? Storage?
- What is the error you are receiving?
In my own use of cloudstack, ive not encountered the type of 'hang' i think you are talking about. Can you share more info? Could be something else in the architecture, setup that is wrong instead? Lets see?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "wangwei0537 (via GitHub)" <gi...@apache.org>.
wangwei0537 commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2689253350
What I mean is that all network features are integrated with CloudStack using physical devices, without using the system virtual machine that comes with CloudStack.Does cloudstack have any related solutions or configuration examples?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2689941587
> Mmm its like this:
>
> VPCs Will use system VMs like the Virtual Router to provide the functionality for VPCs to work. Such as Routing, DHCP, Load Balancing etc
>
> If you dont want to use Virtual Router, you can create L2 Networks using Cloudstack. Its just a simple network that doesnt do anything on its own. So, u will need to have some sort of ‘device’ to provide features such as firewall, routing etc. To do that, Cloudstack has a feature called VNF (Virtual Network Function). It is where you can boot up a VM using an ISO, that had a virtualised network applicance in it. So the most common use case of it is things like PFsense, vyos etc.
>
> It is possible to passthrough hardware appliances to a VM which allows u to have some form of integration with ‘hardware’. But that is not recommended unless u have very specific use cases (and u have to manage it outside of cloudstack). VNF is recommended instead because u can resize the VM according to ur load and save costs. Furthemore, hardware based appliances are pretty expensive… software based can be cheaper and have same performance.
>
> In the cloudstack docs, though it talks about some possible integration with hardwares such as netscalet and palo alto, i highly dont recommend it. It was written maybe 9 years ago and is no longer maintained.
>
> But again it is easier to answer if u share ur specific goal? Like what hardware are u trying to integrate and why?
agree.
there are some network plugins
https://github.com/apache/cloudstack/tree/main/plugins/network-elements
not all of them are well-maintained.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "btzq (via GitHub)" <gi...@apache.org>.
btzq commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2688411049
> Routers, switches, VPN gateways, firewalls and other physical devices
Whats your use case? Cloudstack has L2 Networks and VNF support. It can do some things.
Maybe its easier for u to share what you are trying to achieve?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "dcontiveros (via GitHub)" <gi...@apache.org>.
dcontiveros commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2710611316
Interesting I am currently embarking on this and the lack of openvswitch support is troubling.
I opened a discussions link here to track:
https://github.com/apache/cloudstack/discussions/10523
If support for latest openvswitch is not there on cloudstack then I probably need to start looking elsewhere. The documentation on latest appears that the OVS plugin is extremely limited.
I don't mean to derail the thread though, but I wanted to give my feedback on what I have experienced so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to integrate cloustack with physical network devices to replace system virtual machines [cloudstack]
Posted by "dcontiveros (via GitHub)" <gi...@apache.org>.
dcontiveros commented on issue #10479:
URL: https://github.com/apache/cloudstack/issues/10479#issuecomment-2714245229
OpenVswitch for VLAN tagging and kernel bypass is the goal here.
Essentially i want to ensure I can fully saturate my uplinks with minimal CPU usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org