You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2019/08/26 14:04:08 UTC
[GitHub] [cloudstack] wido opened a new issue #3568: Security Groups are not
properly applied on KVM with multiple NICs
wido opened a new issue #3568: Security Groups are not properly applied on KVM with multiple NICs
URL: https://github.com/apache/cloudstack/issues/3568
##### ISSUE TYPE
* Bug Report
Might be related to #3045
##### COMPONENT NAME
~~~
KVM
Security Groups
~~~
##### CLOUDSTACK VERSION
~~~
4.12
~~~
##### CONFIGURATION
* Advanced Networking
* Instance with NICs in two different networks
##### OS / ENVIRONMENT
Ubuntu 18.04
##### SUMMARY
We have two networks:
Network 1: 10.100.16.0/24
Network 2: 185.109.216.0/24
A Virtual Machine has it's primary NIC in *Network 1* and a Second NIC is attached in *Network 2*.
After a Stop/Start the VM doesn't respond anymore as the improper Security Group rules are applied.
The *nics* part of the StartCommand:
<pre>
[
{
"deviceId": 0,
"networkRateMbps": 5000,
"defaultNic": true,
"pxeDisable": false,
"nicUuid": "9137ceb6-3911-4f58-a027-cd62e9b6d383",
"details": {
"PromiscuousMode": "false",
"MacAddressChanges": "true",
"ForgedTransmits": "true"
},
"dpdkDisabled": false,
"uuid": "ad2f94f1-3b6e-4351-bda7-481195681ab8",
"ip": "10.100.16.228",
"netmask": "255.255.255.0",
"gateway": "10.100.16.1",
"mac": "1e:00:82:00:04:48",
"dns1": "93.180.70.22",
"dns2": "93.180.70.30",
"broadcastType": "Vlan",
"type": "Guest",
"broadcastUri": "vxlan://504",
"isolationUri": "vxlan://504",
"isSecurityGroupEnabled": true,
"ip6address": "2a05:1500:600:1:1c00:82ff:fe00:448",
"ip6cidr": "2a05:1500:600:1::/64"
},
{
"deviceId": 1,
"networkRateMbps": 200,
"defaultNic": false,
"pxeDisable": false,
"nicUuid": "c778c9c4-864b-4fe9-8422-ee8689bf60a4",
"details": {
"PromiscuousMode": "false",
"MacAddressChanges": "true",
"ForgedTransmits": "true"
},
"dpdkDisabled": false,
"uuid": "a0234ec3-bdb6-46b8-80a1-37144d7c3928",
"ip": "185.109.216.175",
"netmask": "255.255.255.0",
"gateway": "185.109.216.3",
"mac": "02:00:21:5b:00:02",
"dns1": "93.180.70.22",
"dns2": "93.180.70.30",
"broadcastType": "Vlan",
"type": "Guest",
"broadcastUri": "vxlan://500",
"isolationUri": "vxlan://500",
"isSecurityGroupEnabled": true,
"ip6address": "2a05:1500:600::21ff:fe5b:2",
"ip6cidr": "2a05:1500:600::/64"
}
]
</pre>
Here you see both NICs and their different networks and VNI.
When the VM is started the Security Groups are applied to the incorrect bridges and vnet devices.
<pre>
2019-08-26 15:12:36,486 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-9:null) (logid:c3238d85) nic=[Nic:Guest-10.100.16.228-vxlan://504]
2019-08-26 15:12:36,486 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-9:null) (logid:c3238d85) Executing: /usr/share/modifyvxlan.sh -v 504 -p vxlan100 -b brvx-504 -o add
</pre>
<pre>
2019-08-26 15:12:36,517 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-9:null) (logid:c3238d85) nic=[Nic:Guest-185.109.216.175-vxlan://500]
2019-08-26 15:12:36,517 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-9:null) (logid:c3238d85) Executing: /usr/share/modifyvxlan.sh -v 500 -p vxlan100 -b brvx-500 -o add
</pre>
So far so good.
VNI 504: 10.100.16.228
VNI 500: 185.109.216.175
The XML definition of the VM now looks like:
~~~
<interface type='bridge'>
<mac address='1e:00:82:00:04:48'/>
<source bridge='brvx-504'/>
<bandwidth>
<inbound average='640000' peak='640000'/>
<outbound average='640000' peak='640000'/>
</bandwidth>
<target dev='vnet13'/>
<model type='virtio'/>
<link state='up'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='02:00:21:5b:00:02'/>
<source bridge='brvx-500'/>
<bandwidth>
<inbound average='25600' peak='25600'/>
<outbound average='25600' peak='25600'/>
</bandwidth>
<target dev='vnet90'/>
<model type='virtio'/>
<link state='up'/>
<alias name='net1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
~~~
Now we see:
| VNI | Address | VNET |
|------|-------------|---------|
| 504 | 10.100.16.228 | vnet13 |
| 500 | 185.109.216.175 | vnet90 |
The KVM Agent now starts to apply the Security Group rules:
<pre>
2019-08-26 15:12:37,243 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-9:null) (logid:c3238d85) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py default_network_rules --vmname i-2-185-VM --vmid 185 --vmip 10.100.16.228 --vmip6 2a05:1500:600:1:1c00:82ff:fe00:448 --vmmac 1e:00:82:00:04:48 --vif vnet13 --brname brvx-504 --nicsecips 0;
</pre>
That looks good! *vnet13* is indeed the proper device for *10.100.16.228*.
It now continues to execute security group rules:
<pre>
2019-08-26 15:12:38,837 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-9:null) (logid:c3238d85) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py default_network_rules --vmname i-2-185-VM --vmid 185 --vmip 185.109.216.175 --vmip6 2a05:1500:600::21ff:fe5b:2 --vmmac 02:00:21:5b:00:02 --vif vnet90 --brname brvx-500 --nicsecips 0;
</pre>
Here the vnet also matches, but it is probably overriding the ipset.
It now starts to apply the SG itself:
<pre>
2019-08-26 15:12:40,853 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-11:null) (logid:18878090) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py add_network_rules --vmname i-2-185-VM --vmid 185 --vmip 185.109.216.175 --vmip6 2a05:1500:600::21ff:fe5b:2 --sig 21f89db6f155e1fd013129e426fb18e7 --seq 4 --vmmac 02:00:21:5b:00:02 --vif vnet13 --brname brvx-504 --nicsecips 0; --rules I:icmp;-1;-1;0.0.0.0/0,NEXT;I:icmp;128;0;::/0,NEXT;I:tcp;1;65535;0.0.0.0/0,::/0,NEXT;I:udp;1;65535;0.0.0.0/0,::/0,NEXT;
</pre>
Here it uses *185.109.216.175* as vmip and it also uses *vnet13* which doesn't match.
The iptable rules now look like this:
<pre>
:i-2-185-VM - [0:0]
:i-2-185-VM-eg - [0:0]
-A i-2-185-VM -p udp -m udp --dport 1:65535 -m state --state NEW -j ACCEPT
-A i-2-185-VM -p tcp -m tcp --dport 1:65535 -m state --state NEW -j ACCEPT
-A i-2-185-VM -p icmp -m icmp --icmp-type any -j ACCEPT
-A i-2-185-VM -j DROP
-A i-2-185-VM-eg -j RETURN
-A i-2-185-def -m physdev --physdev-in vnet90 --physdev-is-bridged -m set ! --match-set i-2-185-VM src -j DROP
-A i-2-185-def -p udp -m physdev --physdev-in vnet90 --physdev-is-bridged -m set --match-set i-2-185-VM src -m udp --dport 53 -j RETURN
-A i-2-185-def -p tcp -m physdev --physdev-in vnet90 --physdev-is-bridged -m set --match-set i-2-185-VM src -m tcp --dport 53 -j RETURN
-A i-2-185-def -m physdev --physdev-in vnet90 --physdev-is-bridged -m set --match-set i-2-185-VM src -j i-2-185-VM-eg
-A i-2-185-def -m physdev --physdev-out vnet90 --physdev-is-bridged -j i-2-185-VM
</pre>
And IPv6
<pre>
:i-2-185-VM - [0:0]
:i-2-185-VM-eg - [0:0]
-A i-2-185-VM -p udp -m udp --dport 1:65535 -m state --state NEW -j ACCEPT
-A i-2-185-VM -p tcp -m tcp --dport 1:65535 -m state --state NEW -j ACCEPT
-A i-2-185-VM -p ipv6-icmp -m icmp6 --icmpv6-type 128/0 -j ACCEPT
-A i-2-185-VM -j DROP
-A i-2-185-VM-eg -j RETURN
-A i-2-185-def -p ipv6-icmp -m physdev --physdev-in vnet90 --physdev-is-bridged -m icmp6 --icmpv6-type 136 -m set --match-set i-2-185-VM-6 src -m hl --hl-eq 255 -j RETURN
-A i-2-185-def -p ipv6-icmp -m physdev --physdev-in vnet90 --physdev-is-bridged -m icmp6 --icmpv6-type 2 -m set --match-set i-2-185-VM-6 src -j RETURN
-A i-2-185-def -p ipv6-icmp -m physdev --physdev-in vnet90 --physdev-is-bridged -m icmp6 --icmpv6-type 1 -m set --match-set i-2-185-VM-6 src -j RETURN
-A i-2-185-def -p ipv6-icmp -m physdev --physdev-in vnet90 --physdev-is-bridged -m icmp6 --icmpv6-type 3 -m set --match-set i-2-185-VM-6 src -j RETURN
-A i-2-185-def -p ipv6-icmp -m physdev --physdev-in vnet90 --physdev-is-bridged -m icmp6 --icmpv6-type 4 -m set --match-set i-2-185-VM-6 src -j RETURN
-A i-2-185-def -p udp -m physdev --physdev-in vnet90 --physdev-is-bridged -m udp --dport 53 -m set --match-set i-2-185-VM-6 src -j RETURN
-A i-2-185-def -p tcp -m physdev --physdev-in vnet90 --physdev-is-bridged -m tcp --dport 53 -m set --match-set i-2-185-VM-6 src -j RETURN
-A i-2-185-def -m physdev --physdev-in vnet90 --physdev-is-bridged -m set ! --match-set i-2-185-VM-6 src -j DROP
-A i-2-185-def -m physdev --physdev-in vnet90 --physdev-is-bridged -m set --match-set i-2-185-VM-6 src -j i-2-185-VM-eg
-A i-2-185-def -m physdev --physdev-out vnet90 --physdev-is-bridged -j i-2-185-VM
</pre>
All the rules have been applied to *vnet90* and that is the device for the second NIC.
The ipset contains the the IPv4 and IPv6 addresses of the second NIC and all the rules are set on *vnet90*.
The same MAC is also used on both NICs while the first NIC has a different NIC.
##### STEPS TO REPRODUCE
Create a VM in Advanced Networking with two NICs on networks which have SG enabled.
##### EXPECTED RESULTS
I would have expected that it would work.
If not I would have expected that the Second NIC had *no* SG applied.
The current implementation is totally broken and renders the VM useless (not working).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services