KVM network flood

[Trevor Francis <tr...@tgrahamcapital.com>]

I have gone through multiple iterations of my setup over the past 2 months with support from the IRC channel and cannot overcome a network flood. I have changed bonding modes, switch ports, complete switches, network topology and anything else I can think of to fix this issue. Here is what happens.

- I currently have CS 3.02 running on KVM hypervisors.
- My host machines have 6 nics. Those nics are split into 2 bonds (management and guest) physical networks. There are 4 bonds in guest and 2 bonds in management. Public and guest traffic go over guest network on 2 separate VLANS and management/storage goes over the management physical.
- The 2 bonds management are split between 2 trunked switches (for fault tolerance at the switch level). The 4 bonds on the guest network are split between 2 trunked switches, so there is a total of 4 switches in my network. The guest network switches have hsrp uplinks to our core router for internet services. 
- I have spanning tree enabled on the switches and on the cloud bridges as well as the VLANs in the switches.
- I have enabled advanced networking.

To allows a private network (guest) and a public network (shared public) to be bound on separate ethernet interfaces for my guest VMs (eth0 and eth1), I create a default isolated network for the guest network and creates a shared public network for the public network. Guest network is on VLAN 11 and public is on VLAN 10.

The storage network works fine, no floods, nothing unusual. (For all intents and purposes the network is configured exactly like the guest network, with 2 less interfaces in the bond).

The isolated guest network works fine as well, nothing unusual. However, when I spin up an instance that I have a public network bound to, the flood begins. It is interesting to note the the flood begins from the VR and not the guest. As soon as the VR is spun up, it starts flooding. If I create a shared network WITHOUT utilizing any services requiring the VR, the network will come up fine...but I have to manually add an IP address to the machine instead of letting the VR assign it through DHCP. I created a new NO for the shared network I am using and only selected DHCP as the services offered....same issue happens with flooding.

I am at my wits end here and dont know how to resolve this. Has anyone else had this issue?




Trevor Francis
Partner
46 Labs | The PeerEdge Cloud
http://www.46labs.com | http://www.peeredge.net
405-362-0046 - Voice  | 405-410-4980 - Cell
trevorgfrancis - Skype
trevor@46labs.com
 
Solutions Provider for the Telecom Industry

 

Re: KVM network flood

[Bryan Whitehead <dr...@megahappy.net>]

On Thu, Nov 1, 2012 at 6:14 AM, Trevor Francis <
trevor.francis@tgrahamcapital.com> wrote:

> I have gone through multiple iterations of my setup over the past 2 months
> with support from the IRC channel and cannot overcome a network flood. I
> have changed bonding modes, switch ports, complete switches, network
> topology and anything else I can think of to fix this issue. Here is what
> happens.
>
> - I currently have CS 3.02 running on KVM hypervisors.
> - My host machines have 6 nics. Those nics are split into 2 bonds
> (management and guest) physical networks. There are 4 bonds in guest and 2
> bonds in management. Public and guest traffic go over guest network on 2
> separate VLANS and management/storage goes over the management physical.
> - The 2 bonds management are split between 2 trunked switches (for fault
> tolerance at the switch level). The 4 bonds on the guest network are split
> between 2 trunked switches, so there is a total of 4 switches in my
> network. The guest network switches have hsrp uplinks to our core router
> for internet services.
> - I have spanning tree enabled on the switches and on the cloud bridges as
> well as the VLANs in the switches.
> - I have enabled advanced networking.
>
>
Each port on the switch needs to be in trunking mode so when a new vlan is
added/created it will work. you can spanning-tree portfast on the ports for
the server(s).

the port interconnecting the switches also need to be in trunking mode so
vlans created can propagate.


> To allows a private network (guest) and a public network (shared public)
> to be bound on separate ethernet interfaces for my guest VMs (eth0 and
> eth1), I create a default isolated network for the guest network and
> creates a shared public network for the public network. Guest network is on
> VLAN 11 and public is on VLAN 10.
>
> The storage network works fine, no floods, nothing unusual. (For all
> intents and purposes the network is configured exactly like the guest
> network, with 2 less interfaces in the bond).
>
> The isolated guest network works fine as well, nothing unusual. However,
> when I spin up an instance that I have a public network bound to, the flood
> begins. It is interesting to note the the flood begins from the VR and not
> the guest. As soon as the VR is spun up, it starts flooding. If I create a
> shared network WITHOUT utilizing any services requiring the VR, the network
> will come up fine...but I have to manually add an IP address to the machine
> instead of letting the VR assign it through DHCP. I created a new NO for
> the shared network I am using and only selected DHCP as the services
> offered....same issue happens with flooding.
>
> I am at my wits end here and dont know how to resolve this. Has anyone
> else had this issue?
>
> I use network bonding myself. What mode are you using for your bonded
interfaces?

However I've simplified mine to just using eth0/eth1. My ifcfg-bond0 looks
has this:
BONDING_OPTS="mode=1 primary=eth0 miimon=1000"

If you are using an alternative bonding mode this has to be fully supported
by your router/switches. I've never had usable success with with mode 2, 3,
5, and 6.

I've found mode 4 requires the bonded nics to be plugged into the same
switch for 802.3ad to correctly work. Maybe your switches are fancy enough
- but I'd take a look at that.

-Bryan