You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Gary Dixon <Ga...@quadris.co.uk.INVALID> on 2022/11/18 14:08:29 UTC
VM instances and RVR cannot communicate with each other when on different hosts
Hi
I am hoping someone could help with a new Dev Cloudstack system we are trying to setup based on Ubuntu 20.04 KVM hosts and mgmt. servers with CS 4.15.2 and an Adv Zone with VPC's
We spotted that the RVR's in the VPC's are both in the 'MASTER' state indicating that they cannot communicate with each other. Also testing within a guest VM - it is able to ping another guest VM in the same network - but only when on the same KVM host.
If we live migrate one of the VM's to a different KVM host then the ping breaks.
Our guest network is using VXLAN isolation method and all network labesl in CS are correct. We are trying to setup this Dev system to match our production system as closely as possible and all networking works perfectly in the Production system
One thing we have noticed is if we put a KVM host into maintenance mode - thus destroying all the brvx-xxx interfaces on there and then bring it back out of maintenance mode and migrate VM's to it and Restart the VPC with cleanup enabled - communication between VM's cross host works again for a few minutes. The RVR's go into Master and Backup status briefly - but then after a few minutes both VR's go into 'MASTER' state and VM's cannot ping each other when on different hosts.
Any pointers/help would be greatly appreciated
BR
Gary
Gary Dixon
Senior Technical Consultant
T: +44 161 537 4990
E: vms@quadris-support.com
W: www.quadris.co.uk
The information contained in this e-mail from Quadris may be confidential and privileged for the private use of the named recipient. The contents of this e-mail may not necessarily represent the official views of Quadris. If you have received this information in error you must not copy, distribute or take any action or reliance on its contents. Please destroy any hard copies and delete this message.
RE: VM instances and RVR cannot communicate with each other when on different hosts
Posted by Alex Mattioli <Al...@shapeblue.com>.
From my experience I had a few issues with RVRs myself, some of those were:
1. They’d fail to elect a master and continuous flip over the connection
2. When going anywhere above 1gbps the Master would fail to send broadcasts messages and the Backup VR would become master, ending up with 2 masters and a messy situation.
3. In ESX, depending on version, you might need to tweak your security parameters for the port group.
Am sure that Andrija has more examples.
Cheers
Alex
From: Gary Dixon <Ga...@quadris.co.uk.INVALID>
Sent: 06 December 2022 15:25
To: users@cloudstack.apache.org
Subject: RE: VM instances and RVR cannot communicate with each other when on different hosts
Thanks for the response Andrija
Do you have any real world examples of the RVR feature 'breaking' or any further info as to why they are not recommended in a production environment /
BR
Gary
Gary Dixon
Senior Technical Consultant
T: +44 161 537 4990
E: v<tel:+44%207989717661>ms@quadris‑support.com
W: www.quadris.co.uk<http://www.quadris.co.uk>
[cid:image784358.png@1C346EC9.9B031216]
The information contained in this e-mail from Quadris may be confidential and privileged for the private use of the named recipient. The contents of this e-mail may not necessarily represent the official views of Quadris. If you have received this information in error you must not copy, distribute or take any action or reliance on its contents. Please destroy any hard copies and delete this message.
-----Original Message-----
From: Andrija Panic <an...@gmail.com>>
Sent: 05 December 2022 12:59
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Subject: Re: VM instances and RVR cannot communicate with each other when on different hosts
Hi,
RVR is a feature that is known to break from time to time, and is NOT recommended for the Production use, at least that's what we advise your customers to do/avoid doing.
However, if 2 VMs can not communicate while on different hosts, but CAN communicate while on the same host - this might indicate VLAN (trunking) misconfiguration on the switches - the VLANs via which those VMs are communicating, has to be trunked to all the hypervisors in your Zone - so that host1 and hostN both can talk over the specific VLAN (here, I'm guessing you are hitting issues with Guest traffic - so check that all your VLANs for Guest traffic are properly trunked on all switch ports to which all of your servers are connected)
Best,
On Fri, 18 Nov 2022 at 15:08, Gary Dixon <Ga...@quadris.co.uk.invalid>>
wrote:
> Hi
>
>
>
> I am hoping someone could help with a new Dev Cloudstack system we are
> trying to setup based on Ubuntu 20.04 KVM hosts and mgmt. servers with
> CS
> 4.15.2 and an Adv Zone with VPC’s
>
>
>
> We spotted that the RVR’s in the VPC’s are both in the ‘MASTER’ state
> indicating that they cannot communicate with each other. Also testing
> within a guest VM – it is able to ping another guest VM in the same
> network – but only when on the same KVM host.
>
> If we live migrate one of the VM’s to a different KVM host then the
> ping breaks.
>
> Our guest network is using VXLAN isolation method and all network
> labesl in CS are correct. We are trying to setup this Dev system to
> match our production system as closely as possible and all networking
> works perfectly in the Production system
>
>
>
> One thing we have noticed is if we put a KVM host into maintenance
> mode – thus destroying all the brvx-xxx interfaces on there and then
> bring it back out of maintenance mode and migrate VM’s to it and
> Restart the VPC with cleanup enabled – communication between VM’s
> cross host works again for a few minutes. The RVR’s go into Master and
> Backup status briefly – but then after a few minutes both VR’s go into
> ‘MASTER’ state and VM’s cannot ping each other when on different hosts.
>
> Any pointers/help would be greatly appreciated
>
>
>
> BR
>
>
>
> Gary
> Gary Dixon
> Senior Technical Consultant
> T: +44 161 537 4990
> E: *v* <+44%207989717661>ms@quadris‑support.com<mailto:ms@quadris%1esupport.com>
> W:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.q
> uadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cefdf1f
> 0e844d4aee130308dad6c081c2%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%
> 7C638058419553464258%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQI
> joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Fq
> bSS%2FHgLkhzM0UTyBY5eH3bepYiwjcvmvIa4%2BWTDds%3D&reserved=0
> The information contained in this e-mail from Quadris may be
> confidential and privileged for the private use of the named
> recipient. The contents of this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents. Please
> destroy any hard copies and delete this message.
>
--
Andrija Panić
RE: VM instances and RVR cannot communicate with each other when on different hosts
Posted by Gary Dixon <Ga...@quadris.co.uk.INVALID>.
Thanks for the response Andrija
Do you have any real world examples of the RVR feature 'breaking' or any further info as to why they are not recommended in a production environment /
BR
Gary
Gary Dixon
Senior Technical Consultant
T: +44 161 537 4990
E: vms@quadris-support.com
W: www.quadris.co.uk
The information contained in this e-mail from Quadris may be confidential and privileged for the private use of the named recipient. The contents of this e-mail may not necessarily represent the official views of Quadris. If you have received this information in error you must not copy, distribute or take any action or reliance on its contents. Please destroy any hard copies and delete this message.
-----Original Message-----
From: Andrija Panic <an...@gmail.com>
Sent: 05 December 2022 12:59
To: users@cloudstack.apache.org
Subject: Re: VM instances and RVR cannot communicate with each other when on different hosts
Hi,
RVR is a feature that is known to break from time to time, and is NOT recommended for the Production use, at least that's what we advise your customers to do/avoid doing.
However, if 2 VMs can not communicate while on different hosts, but CAN communicate while on the same host - this might indicate VLAN (trunking) misconfiguration on the switches - the VLANs via which those VMs are communicating, has to be trunked to all the hypervisors in your Zone - so that host1 and hostN both can talk over the specific VLAN (here, I'm guessing you are hitting issues with Guest traffic - so check that all your VLANs for Guest traffic are properly trunked on all switch ports to which all of your servers are connected)
Best,
On Fri, 18 Nov 2022 at 15:08, Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi
>
>
>
> I am hoping someone could help with a new Dev Cloudstack system we are
> trying to setup based on Ubuntu 20.04 KVM hosts and mgmt. servers with
> CS
> 4.15.2 and an Adv Zone with VPC’s
>
>
>
> We spotted that the RVR’s in the VPC’s are both in the ‘MASTER’ state
> indicating that they cannot communicate with each other. Also testing
> within a guest VM – it is able to ping another guest VM in the same
> network – but only when on the same KVM host.
>
> If we live migrate one of the VM’s to a different KVM host then the
> ping breaks.
>
> Our guest network is using VXLAN isolation method and all network
> labesl in CS are correct. We are trying to setup this Dev system to
> match our production system as closely as possible and all networking
> works perfectly in the Production system
>
>
>
> One thing we have noticed is if we put a KVM host into maintenance
> mode – thus destroying all the brvx-xxx interfaces on there and then
> bring it back out of maintenance mode and migrate VM’s to it and
> Restart the VPC with cleanup enabled – communication between VM’s
> cross host works again for a few minutes. The RVR’s go into Master and
> Backup status briefly – but then after a few minutes both VR’s go into
> ‘MASTER’ state and VM’s cannot ping each other when on different hosts.
>
> Any pointers/help would be greatly appreciated
>
>
>
> BR
>
>
>
> Gary
> Gary Dixon
> Senior Technical Consultant
> T: +44 161 537 4990
> E: *v* <+44%207989717661>ms@quadris‑support.com
> W:
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.q
> uadris.co.uk%2F&data=05%7C01%7CGary.Dixon%40quadris.co.uk%7Cefdf1f
> 0e844d4aee130308dad6c081c2%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%
> 7C638058419553464258%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQI
> joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Fq
> bSS%2FHgLkhzM0UTyBY5eH3bepYiwjcvmvIa4%2BWTDds%3D&reserved=0
> The information contained in this e-mail from Quadris may be
> confidential and privileged for the private use of the named
> recipient. The contents of this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents. Please
> destroy any hard copies and delete this message.
>
--
Andrija Panić
Re: VM instances and RVR cannot communicate with each other when on different hosts
Posted by Andrija Panic <an...@gmail.com>.
Hi,
RVR is a feature that is known to break from time to time, and is NOT
recommended for the Production use, at least that's what we advise your
customers to do/avoid doing.
However, if 2 VMs can not communicate while on different hosts, but CAN
communicate while on the same host - this might indicate VLAN (trunking)
misconfiguration on the switches - the VLANs via which those VMs are
communicating, has to be trunked to all the hypervisors in your Zone - so
that host1 and hostN both can talk over the specific VLAN (here, I'm
guessing you are hitting issues with Guest traffic - so check that all your
VLANs for Guest traffic are properly trunked on all switch ports to which
all of your servers are connected)
Best,
On Fri, 18 Nov 2022 at 15:08, Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi
>
>
>
> I am hoping someone could help with a new Dev Cloudstack system we are
> trying to setup based on Ubuntu 20.04 KVM hosts and mgmt. servers with CS
> 4.15.2 and an Adv Zone with VPC’s
>
>
>
> We spotted that the RVR’s in the VPC’s are both in the ‘MASTER’ state
> indicating that they cannot communicate with each other. Also testing
> within a guest VM – it is able to ping another guest VM in the same network
> – but only when on the same KVM host.
>
> If we live migrate one of the VM’s to a different KVM host then the ping
> breaks.
>
> Our guest network is using VXLAN isolation method and all network labesl
> in CS are correct. We are trying to setup this Dev system to match our
> production system as closely as possible and all networking works perfectly
> in the Production system
>
>
>
> One thing we have noticed is if we put a KVM host into maintenance mode –
> thus destroying all the brvx-xxx interfaces on there and then bring it back
> out of maintenance mode and migrate VM’s to it and Restart the VPC with
> cleanup enabled – communication between VM’s cross host works again for a
> few minutes. The RVR’s go into Master and Backup status briefly – but then
> after a few minutes both VR’s go into ‘MASTER’ state and VM’s cannot ping
> each other when on different hosts.
>
> Any pointers/help would be greatly appreciated
>
>
>
> BR
>
>
>
> Gary
> Gary Dixon
> Senior Technical Consultant
> T: +44 161 537 4990
> E: *v* <+44%207989717661>ms@quadris‑support.com
> W: www.quadris.co.uk
> The information contained in this e-mail from Quadris may be confidential
> and privileged for the private use of the named recipient. The contents of
> this e-mail may not necessarily represent the official views of Quadris.
> If you have received this information in error you must not copy,
> distribute or take any action or reliance on its contents. Please destroy
> any hard copies and delete this message.
>
--
Andrija Panić