You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "TadiosAbebe (via GitHub)" <gi...@apache.org> on 2025/11/01 09:57:48 UTC
[D] Shared network issue on advanced zone with security group [cloudstack]
GitHub user TadiosAbebe created a discussion: Shared network issue on advanced zone with security group
Hello community,
This past few days i was testing shared network in advanced zone with security group and i think i am just missing a lot of understanding about it. and here is my issue
- created a shared network with custom vlan(12) which is out of range of specified for guest network(2048-3072)
- then as soon as i launch an instance on that shared network, other isolated networks connectivity stops working for the physical host where the shared network instance reside in.
- so for example if i have a VR1 and Instance1 in physical host 5, and as soon as the shared network instance gets allocated on physical host 5, the connectivity between Instance1 -> VR1, VR1 -> internet stops working and the same for all others VR and instances on that host
- and I think i boiled down the above issue as being an iptable rules and here is my findings
- iptables -L -n -v listing before creating a shared network and an instance on the shared network
```
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER 0 -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
```
- iptables -L -n -v listing after creating a shared network and an instance on the shared network
```
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 150 packets, 6817 bytes)
pkts bytes target prot opt in out source destination
127 7160 BF-brbond0-12 0 -- * brbond0-12 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-bridged
0 0 BF-brbond0-12 0 -- brbond0-12 * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-bridged
0 0 DROP 0 -- * brbond0-12 0.0.0.0/0 0.0.0.0/0
0 0 DROP 0 -- brbond0-12 * 0.0.0.0/0 0.0.0.0/0
150 6817 DOCKER-USER 0 -- * * 0.0.0.0/0 0.0.0.0/0
150 6817 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain BF-brbond0-12 (2 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
127 7160 BF-brbond0-12-IN 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-in --physdev-is-bridged
121 5168 BF-brbond0-12-OUT 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-out --physdev-is-bridged
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out bond0.12 --physdev-is-bridged
Chain BF-brbond0-12-IN (1 references)
pkts bytes target prot opt in out source destination
6 1992 i-167-1541-def 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged
Chain BF-brbond0-12-OUT (1 references)
pkts bytes target prot opt in out source destination
121 5168 i-167-1541-def 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet8 --physdev-is-bridged
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
150 6817 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
150 6817 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain i-167-1541-VM (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 state NEW
0 0 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain i-167-1541-VM-eg (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0 state NEW
0 0 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain i-167-1541-def (2 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
6 1992 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged udp spt:68 dpt:67
0 0 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet8 --physdev-is-bridged udp spt:67 dpt:68
0 0 DROP 17 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged udp spt:67
0 0 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged ! match-set i-167-1541-VM src
121 5168 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet8 --physdev-is-bridged ! match-set i-167-1541-VM dst
0 0 RETURN 17 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged match-set i-167-1541-VM src udp dpt:53
0 0 RETURN 6 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged match-set i-167-1541-VM src tcp dpt:53
0 0 i-167-1541-VM-eg 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged match-set i-167-1541-VM src
0 0 i-167-1541-VM 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet8 --physdev-is-bridged
```
> As seen in the above output, the dropped packet in `Chain FORWARD (policy DROP 150 packets, 6817 bytes)` keeps just increasing by the seconds afterward. and I don't really understand why isolate network packets are getting dropped once i create a shared network.
- I think the DOCKER related chains in the iptables output are coming from cephadm since i am using compute and storage on the same nodes
I am running apache cloudstack 4.20.1 with ubuntu server 24.04. I would be glad if anyone can shade some light to this.
GitHub link: https://github.com/apache/cloudstack/discussions/11955
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Shared network issue on advanced zone with security group [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: Shared network issue on advanced zone with security group
> I actually enabled security group after the creating of the zone, so when initially launching the zone, it was just advanced zone, the i was able to create isolated network normally as in the docs. then i enabled the security group provider on the physical network when i intend to created shared networks. The isolated networks works just fine until there is an instance on a shared network where they reside on the same physical host. And once the shared network instance is allocated on a host all other network VR->Instance or VR ->internet stops working and only the instance on the shared network works.
>
> @weizhouapache Not sure if that answers your question?
got it, so you are actually using advanced zone , and (1) isolated networks and (2) shared network with security group
right ?
GitHub link: https://github.com/apache/cloudstack/discussions/11955#discussioncomment-14845258
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Shared network issue on advanced zone with security group [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
GitHub user TadiosAbebe added a comment to the discussion: Shared network issue on advanced zone with security group
removed the docker* chains with the following command, but VR->internet and VM->VR connectivity issue still persists
```
iptables -D FORWARD -j DOCKER-USER
iptables -D FORWARD -j DOCKER-ISOLATION-STAGE-1
iptables -F DOCKER
iptables -F DOCKER-USER
iptables -F DOCKER-ISOLATION-STAGE-1
iptables -F DOCKER-ISOLATION-STAGE-2
iptables -X DOCKER
iptables -X DOCKER-USER
iptables -X DOCKER-ISOLATION-STAGE-1
iptables -X DOCKER-ISOLATION-STAGE-2
```
GitHub link: https://github.com/apache/cloudstack/discussions/11955#discussioncomment-14845435
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Shared network issue on advanced zone with security group [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
GitHub user TadiosAbebe edited a comment on the discussion: Shared network issue on advanced zone with security group
I was able to do some more test today spinning up a test Cloudstack environment and here are my findings,
- For a KVM host that doesn't have docker installed, the default FORWARD chain on iptables have an ACCEPT policy, and both instances on isolated network and instances on shared network with security group seems to have connectivity to the outside.
- As soon as docker is installed on that KVM host the default policy for the FORWARD Chain gets changed to DROP and instances on the isolated network loss connectivity to the outside( VM -> VR and VR to internet) while the instance that reside on the shared network with security group works fine.
- I guess the theory here is since shared network with securty groups add its own chain that allow or drop traffic the instance on the shared network keeps working without an issue
- And since isolated networks don't have specific rules the traffic gets caught by the docker DROP policy.
- Adding `iptables -I FORWARD 1 -m physdev --physdev-is-bridged -j ACCEPT` will make instances on the isolated
network to work again.
> It would be great if some one with a better understanding of Linux networking shade some light on the above because i am not sure if this is exactly what is happening it is just my guess. Specifically i don't understand why the docker chain drop the traffic only when a shared network with security group is created on the host.
UPDATTE
> The above happens because, before creating a shared network with security group vm traffic such as bridged L2 forwarding does not necessarily traverse iptables. bridged traffic can bypass iptables unless bridge netfilter is enabled. which is is controlled by the br_netfilter module + sysctls. when you create a shared network with SG on cloudstack it makes bridged guest traffic start traversing the host’s FORWARD chain, and that’s when Docker’s “FORWARD policy = DROP” becomes fatal for networks that don’t have explicit allow rules.
Anyways for my solution, instead of adding `iptables -I FORWARD 1 -m physdev --physdev-is-bridged -j ACCEPT`, When I review the Cloudstack installation guide and I saw steps on setting the DEFAULT_FORWARD_POLICY to ACCEPT and enabling ufw, when i check my ufw it was inactive(don't remember why i have disabled it), but I add ufw rules and re-enabled ufw. Not sure if what resolved the connectivity issue is the DEFAULT_FORWARD_POLICY or other ufw rules, but it resolved the issue, now both the instances on the shared network and isolated network can communicate out to the internet.
Now getting back to my followup questions
1. the reason i needed to create a shared network is mainly for public traffic for guest vms that need direct public ip to virtual machine without a VR, and i am only able to create the shared network on physical network 3(which is the bridge that have guest traffic type), will i be having any issue architecturally residing public traffic on guest traffic type?
2. what issue would i be having if i create different shared network with the same vid(by enabling vlan id overlap) for different domains and set different start and end ip range. So for example
- domain1 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.5 end_ip: 172.19.1.10]
- domain2 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.11 end_ip: 172.19.1.15]
- domain3 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.16 end_ip: 172.19.1.20]
from my understanding if i do the above, user on domain1 can only use ip 172.19.1.5-172.19.1.10 securiy groups will prevent him from stealing and manually assigning an ip from domain2. is this correct?
GitHub link: https://github.com/apache/cloudstack/discussions/11955#discussioncomment-14847348
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Shared network issue on advanced zone with security group [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: Shared network issue on advanced zone with security group
> actually, here is what i did, there were a bunch of isolated networks and vm, i have 5 nodes in my zone all on the same cluster. so one day when i am testing the shared network scenario, all the vms and isolated network VRs stops working, i first thought i created a vlan conflict where i affected the other resources, but when i dig down i saw that the only affected vms and isolated network VRs are the one that resides on the same physical host as the one that have the vm on the shared network. then i migrated all isolated network VRs and instances from that host. So the above iptable rules are from one of the host where there is 1 isolated network VR, 1 instance on that isolated network and 1 instance on a shared network with security group(without VR)
Try out with removing the DOCKER* chains
GitHub link: https://github.com/apache/cloudstack/discussions/11955#discussioncomment-14845376
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Shared network issue on advanced zone with security group [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
GitHub user TadiosAbebe added a comment to the discussion: Shared network issue on advanced zone with security group
The thing is, I couldn't not say the docker chain is dropping the traffic because that chain is on all the other hosts and it is not affecting them
One thing i know for sure right now is that, the security group iptables rules are interfering with the isolated network traffic. cause i created a new service offering without a security group and launching a vm on that security group doesn't interrupt existing isolated network traffic on the same host.
On a similar not i have a follow up question.
1. the reason i needed to create a shared network is mainly for public traffic for guest vms that need direct public ip to virtual machine without a VR, and i am only able to create the shared network on physical network 3(which is the bridge that have guest traffic type), will i be having any issue architecturally residing public traffic on guest traffic type
2. what issue would i be having if i create different shared network with the same vid(by enabling vlan id overlap) for different domains and set different start and end ip range. So for example
- domain1 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.5 end_ip: 172.19.1.10]
- domain2 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.11 end_ip: 172.19.1.15]
- domain3 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.16 end_ip: 172.19.1.20]
- from my understanding if i do the above, user on domain1 can only use ip 172.19.1.5-172.19.1.10 securiy groups will prevent him from stealing and manually assigning an ip from domain2. is this correct?
GitHub link: https://github.com/apache/cloudstack/discussions/11955#discussioncomment-14845753
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Shared network issue on advanced zone with security group [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Shared network issue on advanced zone with security group
@TadiosAbebe can you update on the status of this question? Did you continue investigating and do you have more to share or should we close it?
GitHub link: https://github.com/apache/cloudstack/discussions/11955#discussioncomment-17809710
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Shared network issue on advanced zone with security group [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
GitHub user TadiosAbebe added a comment to the discussion: Shared network issue on advanced zone with security group
I actually enabled security group after the creating of the zone, so when initially launching the zone, it was just advanced zone, the i was able to create isolated network normally as in the docs. then i enabled the security group provider on the physical network when i intend to created shared networks. The isolated networks works just fine until there is an instance on a shared network where they reside on the same physical host.
And once the shared network instance is allocated on a host all other network VR->Instance or VR ->internet stops working and only the instance on the shared network works.
@weizhouapache Not sure if that answers your question?
GitHub link: https://github.com/apache/cloudstack/discussions/11955#discussioncomment-14845214
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Shared network issue on advanced zone with security group [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: Shared network issue on advanced zone with security group
just curious, how to create isolated network in advanced zone with security groups ?
GitHub link: https://github.com/apache/cloudstack/discussions/11955#discussioncomment-14845172
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Shared network issue on advanced zone with security group [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
GitHub user TadiosAbebe added a comment to the discussion: Shared network issue on advanced zone with security group
I was able to do some more test today spinning up a test Cloudstack environment and here are my findings,
- For a KVM host that doesn't have docker installed, the default FORWARD chain on iptables have an ACCEPT policy, and both instances on isolated network and instances on shared network with security group seems to have connectivity to the outside.
- As soon as docker is installed on that KVM host the default policy for the FORWARD Chain gets changed to DROP and instances on the isolated network loss connectivity to the outside( VM -> VR and VR to internet) while the instance that reside on the shared network with security group works fine.
- I guess the theory here is since shared network with securty groups add its own chain that allow or drop traffic the instance on the shared network keeps working without an issue
- And since isolated networks don't have specific rules the traffic gets caught by the docker DROP policy.
- Adding `iptables -I FORWARD 1 -m physdev --physdev-is-bridged -j ACCEPT` will make instances on the isolated
network to work again.
> It would be great if some one with a better understanding of Linux networking shade some light on the above because i am not sure if this is exactly what is happening it is just my guess. Specifically i don't understand why the docker chain drop the traffic only when a shared network with security group is created on the host.
Anyways for my solution, instead of adding `iptables -I FORWARD 1 -m physdev --physdev-is-bridged -j ACCEPT`, When I review the Cloudstack installation guide and I saw steps on setting the DEFAULT_FORWARD_POLICY to ACCEPT and enabling ufw, when i check my ufw it was inactive(don't remember why i have disabled it), but I add ufw rules and re-enabled ufw. Not sure if what resolved the connectivity issue is the DEFAULT_FORWARD_POLICY or other ufw rules, but it resolved the issue, now both the instances on the shared network and isolated network can communicate out to the internet.
Now getting back to my followup questions
1. the reason i needed to create a shared network is mainly for public traffic for guest vms that need direct public ip to virtual machine without a VR, and i am only able to create the shared network on physical network 3(which is the bridge that have guest traffic type), will i be having any issue architecturally residing public traffic on guest traffic type?
2. what issue would i be having if i create different shared network with the same vid(by enabling vlan id overlap) for different domains and set different start and end ip range. So for example
- domain1 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.5 end_ip: 172.19.1.10]
- domain2 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.11 end_ip: 172.19.1.15]
- domain3 -> share-net-domain1[vid: 12 gw: 172.19.1.1 netmask: 255.255.255.0 start_ip: 172.19.1.16 end_ip: 172.19.1.20]
from my understanding if i do the above, user on domain1 can only use ip 172.19.1.5-172.19.1.10 securiy groups will prevent him from stealing and manually assigning an ip from domain2. is this correct?
GitHub link: https://github.com/apache/cloudstack/discussions/11955#discussioncomment-14847348
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Shared network issue on advanced zone with security group [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
GitHub user TadiosAbebe added a comment to the discussion: Shared network issue on advanced zone with security group
actually, here is what i did, there were a bunch of isolated networks and vm, i have 5 nodes in my zone all on the same cluster. so one day when i am testing the shared network scenario, all the vms and isolated network VRs stops working, i first thought i created a vlan conflict where i affected the other resources, but when i dig down i saw that the only affected vms and isolated network VRs are the one that resides on the same physical host as the one that have the vm on the shared network. then i migrated all isolated network VRs and instances from that host. So the above iptable rules are from one of the host where there is 1 isolated network VR, 1 instance on that isolated network and 1 instance on a shared network with security group(without VR)
GitHub link: https://github.com/apache/cloudstack/discussions/11955#discussioncomment-14845310
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Shared network issue on advanced zone with security group [cloudstack]
Posted by "TadiosAbebe (via GitHub) via users" <us...@cloudstack.apache.org>.
GitHub user TadiosAbebe closed a discussion: Shared network issue on advanced zone with security group
Hello community,
This past few days i was testing shared network in advanced zone with security group and i think i am just missing a lot of understanding about it. and here is my issue
- created a shared network with custom vlan(12) which is out of range of specified for guest network(2048-3072)
- then as soon as i launch an instance on that shared network, other isolated networks connectivity stops working for the physical host where the shared network instance reside in.
- so for example if i have a VR1 and Instance1 in physical host 5, and as soon as the shared network instance gets allocated on physical host 5, the connectivity between Instance1 -> VR1, VR1 -> internet stops working and the same for all others VR and instances on that host
- and I think i boiled down the above issue as being an iptable rules and here is my findings
- iptables -L -n -v listing before creating a shared network and an instance on the shared network
```
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER 0 -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
```
- iptables -L -n -v listing after creating a shared network and an instance on the shared network
```
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 150 packets, 6817 bytes)
pkts bytes target prot opt in out source destination
127 7160 BF-brbond0-12 0 -- * brbond0-12 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-bridged
0 0 BF-brbond0-12 0 -- brbond0-12 * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-bridged
0 0 DROP 0 -- * brbond0-12 0.0.0.0/0 0.0.0.0/0
0 0 DROP 0 -- brbond0-12 * 0.0.0.0/0 0.0.0.0/0
150 6817 DOCKER-USER 0 -- * * 0.0.0.0/0 0.0.0.0/0
150 6817 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 0 -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain BF-brbond0-12 (2 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
127 7160 BF-brbond0-12-IN 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-in --physdev-is-bridged
121 5168 BF-brbond0-12-OUT 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-out --physdev-is-bridged
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out bond0.12 --physdev-is-bridged
Chain BF-brbond0-12-IN (1 references)
pkts bytes target prot opt in out source destination
6 1992 i-167-1541-def 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged
Chain BF-brbond0-12-OUT (1 references)
pkts bytes target prot opt in out source destination
121 5168 i-167-1541-def 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet8 --physdev-is-bridged
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
150 6817 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
150 6817 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain i-167-1541-VM (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 state NEW
0 0 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain i-167-1541-VM-eg (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0 state NEW
0 0 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain i-167-1541-def (2 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
6 1992 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged udp spt:68 dpt:67
0 0 ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet8 --physdev-is-bridged udp spt:67 dpt:68
0 0 DROP 17 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged udp spt:67
0 0 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged ! match-set i-167-1541-VM src
121 5168 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet8 --physdev-is-bridged ! match-set i-167-1541-VM dst
0 0 RETURN 17 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged match-set i-167-1541-VM src udp dpt:53
0 0 RETURN 6 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged match-set i-167-1541-VM src tcp dpt:53
0 0 i-167-1541-VM-eg 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet8 --physdev-is-bridged match-set i-167-1541-VM src
0 0 i-167-1541-VM 0 -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet8 --physdev-is-bridged
```
> As seen in the above output, the dropped packet in `Chain FORWARD (policy DROP 150 packets, 6817 bytes)` keeps just increasing by the seconds afterward. and I don't really understand why isolate network packets are getting dropped once i create a shared network.
- I think the DOCKER related chains in the iptables output are coming from cephadm since i am using compute and storage on the same nodes
I am running apache cloudstack 4.20.1 with ubuntu server 24.04. I would be glad if anyone can shade some light to this.
GitHub link: https://github.com/apache/cloudstack/discussions/11955
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org