You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "dstoy53 (via GitHub)" <gi...@apache.org> on 2025/06/06 04:16:01 UTC
[D] one to many secondary IP for a floating VIP with SG [cloudstack]
GitHub user dstoy53 created a discussion: one to many secondary IP for a floating VIP with SG
Hey folks,
I'm experimenting with an advanced zone with security groups enabled in a shared network, and I'm trying to accommodate a keepalived VIP.
The traffic to the VIP only succeeds if it's assigned as a secondary IP to the active VM so that it gets added to iptables + ebtables. I've validated this behavior by failing over and then manually adding the VIP to the ip set and to the relevant ebtables chain. Trying to add the same secondary IP to the second VM via ui/api fails.
Without looking at everything else this idea might break, I think the easiest solution would be to allow a secondary IP to be attached to multiple VMs so the unmodified security group script can handle the rules.
Am I missing something obvious that could more easily solve this use case?
I also ran a small test that appeared to succeed:
1. for the first VM, add a secondary IP to act as the VIP
2. for the second VM, clone the relevant row in `nic_secondary_ips` replacing the vmId and nicId with those belonging to the 2nd VM (and satisfying the uuid constraint)
3. for the second VM, in the `nics` table toggle secondary_ip to 1
4. perform some live migrations to trigger security_groups.py on the target hosts
The VIP appears to fail over as expected when I start/stop keepalived in the VMs to trigger failover. I also checked iptables/ebtables and everything looks good there with the primary/secondary IPs for both VMs.
The UI shows the same secondary IP in each VM's NIC settings. The address list in Guest Networks shows the IP as allocated, though if I refresh the page it cycles between showing instance 01 vs 02.
Previous discussions on this topic tend to suggest modifying security_groups.py but I think this might provide a cleaner answer unless it breaks something else.
GitHub link: https://github.com/apache/cloudstack/discussions/10979
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] one to many secondary IP for a floating VIP with SG [cloudstack]
Posted by "dstoy53 (via GitHub)" <gi...@apache.org>.
GitHub user dstoy53 added a comment to the discussion: one to many secondary IP for a floating VIP with SG
I agree that some type of flag would be good for safety, otherwise an api user won't know if they're allocating a shared secondary IP without first checking the allocation status. The UI would already recognize the IP as allocated and hide it on the NIC page, so that might need some adjustments to enable this behavior.
I think kubernetes clusters with a cni managed overlay will work well enough since outbound traffic would get a SNAT with the primary VM IP of each node. In this scenario the inbound traffic would need a load balancer with a VIP for the control plane nodes and a VIP for an ingress controller (or other nodeport services), so only the VIPs would need to be able to float.
Attaching pods directly to the network would get more complicated and the CNI would need to talk to the cloudstack api to attach/detach IPs. This is what cilium does in aws. In AWS there's also prefix delegation, because if the CNI keeps requesting /32s for each pod you hit a maximum on the ENI, and if it keeps adding ENIs you hit the maximum ENIs for the instance type. Prefix delegation allocates a /28 to the ENI (consuming the same capacity as a single /32 on the ENI) and pod addresses are assigned from that prefix.
GitHub link: https://github.com/apache/cloudstack/discussions/10979#discussioncomment-13392195
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] one to many secondary IP for a floating VIP with SG [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: one to many secondary IP for a floating VIP with SG
> I agree that some type of flag would be good for safety, otherwise an api user won't know if they're allocating a shared secondary IP without first checking the allocation status. The UI would already recognize the IP as allocated and hide it on the NIC page, so that might need some adjustments to enable this behavior.
>
> I think kubernetes clusters with a cni managed overlay will work well enough since outbound traffic would get a SNAT with the primary VM IP of each node. In this scenario the inbound traffic would need a load balancer with a VIP for the control plane nodes and a VIP for an ingress controller (or other nodeport services), so only the VIPs would need to be able to float.
>
> Attaching pods directly to the network would get more complicated and the CNI would need to talk to the cloudstack api to attach/detach IPs. This is what cilium does in aws. In AWS there's also prefix delegation, because if the CNI keeps requesting /32s for each pod you hit a maximum on the ENI, and if it keeps adding ENIs you hit the maximum ENIs for the instance type. Prefix delegation allocates a /28 to the ENI (consuming the same capacity as a single /32 on the ENI) and pod addresses are assigned from that prefix.
thanks for the sharing @dstoy53
Allocating /28 subnet to the ENI seems a good idea.
I do not know how security groups works on the IPs (maybe allow all traffic on hypervisor level and let ingress controller to manage ?)
GitHub link: https://github.com/apache/cloudstack/discussions/10979#discussioncomment-13726424
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] one to many secondary IP for a floating VIP with SG [cloudstack]
Posted by "rajujith (via GitHub)" <gi...@apache.org>.
GitHub user rajujith deleted a comment on the discussion: one to many secondary IP for a floating VIP with SG
good workaround @dstoy53
GitHub link: https://github.com/apache/cloudstack/discussions/10979#discussioncomment-13387108
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] one to many secondary IP for a floating VIP with SG [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: one to many secondary IP for a floating VIP with SG
it looks like a very good workaround.
we could add a flag (e.g. vip) to secondary ip . this kind of IP can be associated to multiple vms under same account.
just curious, anyone has used kubernetes clusters or containers inside vms on shared networks with security groups ?
GitHub link: https://github.com/apache/cloudstack/discussions/10979#discussioncomment-13386608
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] one to many secondary IP for a floating VIP with SG [cloudstack]
Posted by "rajujith (via GitHub)" <gi...@apache.org>.
GitHub user rajujith added a comment to the discussion: one to many secondary IP for a floating VIP with SG
good workaround @dstoy53
GitHub link: https://github.com/apache/cloudstack/discussions/10979#discussioncomment-13387108
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] one to many secondary IP for a floating VIP with SG [cloudstack]
Posted by "rajujith (via GitHub)" <gi...@apache.org>.
GitHub user rajujith added a comment to the discussion: one to many secondary IP for a floating VIP with SG
Good workaround @dstoy53
GitHub link: https://github.com/apache/cloudstack/discussions/10979#discussioncomment-13387113
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org