You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Andrei Mikhailovsky <an...@arhont.com.INVALID> on 2018/02/21 12:27:03 UTC
VR routing issues in Advanced Mode
Hello
Could someone help me to identify the routing issues that we have. The problem is the traffic from different guest networks can not reach each other via the public IPs.
Here is my ACS setup:
ACS 4.9.3.0 (both management and agents)
KVM Hypervisor based on Ubuntu 16.04
Ceph as primary storage. NFS as secondary storage
Advanced Networking with vlan separation
2 x Public IP ranges with /26 netmask.
Here is an example when routing DOES NOT work:
Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide all networking services (dhcp, fw, port forwarding, load balancing, etc)
Guest Network 1:
Public IP: XXX.XXX.XXX.10/26
Private IP range: 10.1.1.0/24
guest vm1 IP: 10.1.1.100/24
Guest Network 2:
Public IP: XXX.XXX.XXX.20/26
Private IP range: 10.1.1.0/24
guest vm2 IP: 10.1.1.200/24
I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port 80. I've created the port forwarding rules to forward port 80 from public XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200 respectively.
This setup works perfectly well when I am initiating the connections from outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP XXX.XXX.XXX.20.
Here is an example when the routing DOES work:
Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are given directly to a guest vm
Guest Network 1:
guest vm1 Public IP: XXX.XXX.XXX.100/26
Guest Network 2:
guest vm2 Public IP: XXX.XXX.XXX.110/26
In the Case 2, the guest vm has a public IP address directly assigned to its network interface. VRs are not used for this networking. Each guest has a fw rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can access each other on port 80. Also, vms from Case 1 above can access port 80 on vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case 1.
So, it seems that the rules on the VR in Case 1 do not allow traffic that originates from other VRs within the same public network range. The trace route shows the last hop being the VR's private IP address. How do I change that behaviour and fix the networking issue?
Thanks
Andrei
Re: VR routing issues in Advanced Mode
Posted by Andrei Mikhailovsky <an...@arhont.com.INVALID>.
Hi Dag,
Sorry for not being clear.
The ICMP works just fine. so the VPC network to VPC network ICMP works. However, from what I can see, the ping is replied by the VR itself without forwarding traffic to the virtual machine. So, the VPC virtual routers can see each other pings. The problem is with port forwarding to the vms.
I will investigate on the VR side to see if they are actually forwarding tcp packets to the vm.
Cheers
----- Original Message -----
> From: "Dag Sonstebo" <Da...@shapeblue.com>
> To: "users" <us...@cloudstack.apache.org>
> Sent: Tuesday, 27 February, 2018 18:58:44
> Subject: Re: VR routing issues in Advanced Mode
> Hi Andrei,
>
> Sorry lost you - are you saying it's now all working when you allow icmp in your
> ACLs?
>
> If not can you look at the tcpdumps on source and destination VRs as per my
> previous post? You may obviously have to run these on different interfaces
> depending on which VPC tier you are pinging from.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
> S: +44 20 3603 0540 | dag.sonstebo@shapeblue.com | http://www.shapeblue.com
> <http://www.shapeblue.com/> | Twitter:@ShapeBlue
> <https://twitter.com/#!/shapeblue>
>
>
>On 27/02/2018, 10:27, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
>
> Hi Dag,
>
> Thanks, for your reply which I've missed earlier.
>
> I have done some more digging around and would like to make some corrections to
> the problem at hand.
>
> 1. It seems that the problem only effects VPC networks within cloudstack.
> 2. Networks which use non-VPC networking can talk to each other.
> 3. VPC to VPC traffic is not working.
> 4. VPC traffic can reach non-VPC network
> 5. non-VPC traffic can't reach VPC network
>
> In all cases, if the icmp is allowed in the ACLs, all networks can ping each
> other. So, the traffic is being routed and reaches the virtual router.
>
> Any advice?
>
> Thanks
>
>
>
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
> ----- Original Message -----
> > From: "Dag Sonstebo" <Da...@shapeblue.com>
> > To: "users" <us...@cloudstack.apache.org>
> > Sent: Friday, 23 February, 2018 16:45:04
> > Subject: Re: VR routing issues in Advanced Mode
>
> > Hi Andrei,
> >
> > Next step is to do some tcpdumping on the two VRs. Set some ping’s going and
> > check:
> >
> > Private NIC: tcpdump -i eth0 icmp
> > Public NIC: tcpdump -i eth2 icmp
> >
> > That way you should be able to see how far your traffic reaches.
> >
> >
> > Regards,
> > Dag Sonstebo
> > Cloud Architect
> > ShapeBlue
> >
> > On 23/02/2018, 16:17, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
> >
> > Bump.
> >
> > Any ideas anyone? This issue is really annoying.
> >
> > Thanks
> >
> >
> > Dag.Sonstebo@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> > ----- Original Message -----
> > > From: "Andrei Mikhailovsky" <an...@arhont.com.INVALID>
> > > To: "users" <us...@cloudstack.apache.org>, "users" <us...@cloudstack.apache.org>
> > > Sent: Wednesday, 21 February, 2018 22:10:25
> > > Subject: Re: VR routing issues in Advanced Mode
> >
> > > Dag,
> > >
> > >
> > >
> > >
> > > Yeah, we have egress traffic enabled. We also use VPCs on some of the networks
> > > and they are also effected by this issue along with None VPC networks.
> > >
> > >
> > >
> > >
> > > Any thoughts?
> > >
> > >
> > >
> > >
> > > Andrei Mikhailovsky
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: Dag Sonstebo
> > >
> > >
> > > Sent: Wednesday 21 February, 18:30
> > >
> > >
> > > Subject: Re: VR routing issues in Advanced Mode
> > >
> > >
> > > To: users@cloudstack.apache.org
> > >
> > >
> > >
> > >
> > >
> > >
> > > Hi Andrei,
> > >
> > >
> > >
> > >
> > > Understand. To get all the obvious things out the way – have you allowed egress
> > > traffic on the two networks (you mention ACLs which we only use on VPCs and
> > > basic networks)?
> > >
> > >
> > >
> > >
> > > Regards,
> > >
> > >
> > > Dag Sonstebo
> > >
> > >
> > > Cloud Architect
> > >
> > >
> > > ShapeBlue
> > >
> > >
> > >
> > >
> > > On 21/02/2018, 14:51, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
> > >
> > >
> > >
> > >
> > > Hi Dag,
> > >
> > >
> > >
> > >
> > > Please see my comments below:
> > >
> > >
> > >
> > >
> > >> Hi Andrei,
> > >
> > >
> > >>
> > >
> > >
> > >> You’re confusing the matter with your masking of public IP ranges. You said you
> > >
> > >
> > >> have “2 x Public IP ranges with /26 netmask” – but since you are masking them
> > >
> > >
> > >> out with X’s your email doesn’t make sense. If all the X’s are the same then a
> > >
> > >
> > >> .10 and a .20 IP address would be on the same /26 network.
> > >
> > >
> > >>
> > >
> > >
> > >> I will assume that you do in fact have 2 x 26-bit networks, e.g.:
> > >
> > >
> > >>
> > >
> > >
> > >> 192.168.0.0/26 – with default gateway 192.168.0.1
> > >
> > >
> > >> 192.168.0.64/26 – with default gateway 192.168.0.65
> > >
> > >
> > >>
> > >
> > >
> > >
> > >
> > >
> > >
> > > That is correct. I do have two separate /26 networks similar to what you've
> > > described above. However, one /26 is used for direct public IP service
> > > offering, where VRs are not involved in networking at all and the second /26 is
> > > used for the service offering where VRs are used to provide the networking
> > > function.
> > >
> > >
> > >
> > >
> > >
> > >
> > >> If your two guest networks have VRs on separate public IP ranges you will have
> > >
> > >
> > >> e.g.
> > >
> > >
> > >>
> > >
> > >
> > >> VR1: public IP 192.168.0.10
> > >
> > >
> > >> VR2: public IP 192.168.0.70
> > >
> > >
> > >>
> > >
> > >
> > >
> > >
> > >
> > >
> > > Nope, the guest vms with VRs that can't talk to each other are on the same /26
> > > network. (in your example that would be on the same 192.168.0.0/26)
> > >
> > >
> > >
> > >
> > >
> > >
> > >> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up
> > >
> > >
> > >> routing and possibly firewalling on the data centre device which handles the
> > >
> > >
> > >> default gateway for the two networks – i.e. the top of rack switch or router
> > >
> > >
> > >> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you
> > >
> > >
> > >> can reach services on both networks from outside this range makes sense.
> > >
> > >
> > >>
> > >
> > >
> > >
> > >
> > > These has been set up and vms between separate /26 networks CAN talk to each
> > > other. The VMs on the same /26 network that doesn't use the VR service can also
> > > talk to each other. The problem with VMs on the same /26 that use VRs can't
> > > talk to each other using their public IP addresses.
> > >
> > >
> > >
> > >
> > >
> > >
> > >> So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER > VR2
> > >
> > >
> > >> > VM2.
> > >
> > >
> > >>
> > >
> > >
> > >>
> > >
> > >
> > >> Regards,
> > >
> > >
> > >> Dag Sonstebo
> > >
> > >
> > >> Cloud Architect
> > >
> > >
> > >> ShapeBlue
> > >
> > >
> > >>
> > >
> > >
> > >> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
> > >
> > >
> > >>
> > >
> > >
> > >> Hello
> > >
> > >
> > >>
> > >
> > >
> > >> Could someone help me to identify the routing issues that we have. The problem
> > >
> > >
> > >> is the traffic from different guest networks can not reach each other via the
> > >
> > >
> > >> public IPs.
> > >
> > >
> > >>
> > >
> > >
> > >> Here is my ACS setup:
> > >
> > >
> > >> ACS 4.9.3.0 (both management and agents)
> > >
> > >
> > >> KVM Hypervisor based on Ubuntu 16.04
> > >
> > >
> > >> Ceph as primary storage. NFS as secondary storage
> > >
> > >
> > >> Advanced Networking with vlan separation
> > >
> > >
> > >> 2 x Public IP ranges with /26 netmask.
> > >
> > >
> > >>
> > >
> > >
> > >>
> > >
> > >
> > >>
> > >
> > >
> > >> Here is an example when routing DOES NOT work:
> > >
> > >
> > >>
> > >
> > >
> > >> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide
> > >
> > >
> > >> all networking services (dhcp, fw, port forwarding, load balancing, etc)
> > >
> > >
> > >>
> > >
> > >
> > >> Guest Network 1:
> > >
> > >
> > >>
> > >
> > >
> > >> Public IP: XXX.XXX.XXX.10/26
> > >
> > >
> > >> Private IP range: 10.1.1.0/24
> > >
> > >
> > >> guest vm1 IP: 10.1.1.100/24
> > >
> > >
> > >>
> > >
> > >
> > >> Guest Network 2:
> > >
> > >
> > >> Public IP: XXX.XXX.XXX.20/26
> > >
> > >
> > >> Private IP range: 10.1.1.0/24
> > >
> > >
> > >> guest vm2 IP: 10.1.1.200/24
> > >
> > >
> > >>
> > >
> > >
> > >>
> > >
> > >
> > >> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port
> > >
> > >
> > >> 80. I've created the port forwarding rules to forward port 80 from public
> > >
> > >
> > >> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
> > >
> > >
> > >> respectively.
> > >
> > >
> > >>
> > >
> > >
> > >> This setup works perfectly well when I am initiating the connections from
> > >
> > >
> > >> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the
> > >
> > >
> > >> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP
> > >
> > >
> > >> XXX.XXX.XXX.20.
> > >
> > >
> > >>
> > >
> > >
> > >>
> > >
> > >
> > >>
> > >
> > >
> > >>
> > >
> > >
> > >> Here is an example when the routing DOES work:
> > >
> > >
> > >>
> > >
> > >
> > >> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are
> > >
> > >
> > >> given directly to a guest vm
> > >
> > >
> > >>
> > >
> > >
> > >> Guest Network 1:
> > >
> > >
> > >>
> > >
> > >
> > >> guest vm1 Public IP: XXX.XXX.XXX.100/26
> > >
> > >
> > >>
> > >
> > >
> > >> Guest Network 2:
> > >
> > >
> > >>
> > >
> > >
> > >> guest vm2 Public IP: XXX.XXX.XXX.110/26
> > >
> > >
> > >>
> > >
> > >
> > >> In the Case 2, the guest vm has a public IP address directly assigned to its
> > >
> > >
> > >> network interface. VRs are not used for this networking. Each guest has a fw
> > >
> > >
> > >> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can
> > >
> > >
> > >> access each other on port 80. Also, vms from Case 1 above can access port 80 on
> > >
> > >
> > >> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case
> > >
> > >
> > >> 1.
> > >
> > >
> > >>
> > >
> > >
> > >>
> > >
> > >
> > >>
> > >
> > >
> > >> So, it seems that the rules on the VR in Case 1 do not allow traffic that
> > >
> > >
> > >> originates from other VRs within the same public network range. The trace route
> > >
> > >
> > >> shows the last hop being the VR's private IP address. How do I change that
> > >
> > >
> > >> behaviour and fix the networking issue?
> > >
> > >
> > >>
> > >
> > >
> > >> Thanks
> > >
> > >
> > >>
> > >
> > >
> > >> Andrei
> > >
> > >
> > >>
> > >
> > >
> > >>
> > >
> > >
> > >>
> > >
> > >
> > >> Dag.Sonstebo@shapeblue.com
> > >
> > >
> > >> www.shapeblue.com
> > >
> > >
> > >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > >
> > >
> > >> @shapeblue
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Dag.Sonstebo@shapeblue.com
> > >
> > >
> > > www.shapeblue.com
> > >
> > >
> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > >
> > >
> > > @shapeblue
Re: VR routing issues in Advanced Mode
Posted by Dag Sonstebo <Da...@shapeblue.com>.
Hi Andrei,
Sorry lost you - are you saying it's now all working when you allow icmp in your ACLs?
If not can you look at the tcpdumps on source and destination VRs as per my previous post? You may obviously have to run these on different interfaces depending on which VPC tier you are pinging from.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
S: +44 20 3603 0540 | dag.sonstebo@shapeblue.com | http://www.shapeblue.com <http://www.shapeblue.com/> | Twitter:@ShapeBlue <https://twitter.com/#!/shapeblue>
On 27/02/2018, 10:27, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
Hi Dag,
Thanks, for your reply which I've missed earlier.
I have done some more digging around and would like to make some corrections to the problem at hand.
1. It seems that the problem only effects VPC networks within cloudstack.
2. Networks which use non-VPC networking can talk to each other.
3. VPC to VPC traffic is not working.
4. VPC traffic can reach non-VPC network
5. non-VPC traffic can't reach VPC network
In all cases, if the icmp is allowed in the ACLs, all networks can ping each other. So, the traffic is being routed and reaches the virtual router.
Any advice?
Thanks
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
----- Original Message -----
> From: "Dag Sonstebo" <Da...@shapeblue.com>
> To: "users" <us...@cloudstack.apache.org>
> Sent: Friday, 23 February, 2018 16:45:04
> Subject: Re: VR routing issues in Advanced Mode
> Hi Andrei,
>
> Next step is to do some tcpdumping on the two VRs. Set some ping’s going and
> check:
>
> Private NIC: tcpdump -i eth0 icmp
> Public NIC: tcpdump -i eth2 icmp
>
> That way you should be able to see how far your traffic reaches.
>
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 23/02/2018, 16:17, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
>
> Bump.
>
> Any ideas anyone? This issue is really annoying.
>
> Thanks
>
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
> ----- Original Message -----
> > From: "Andrei Mikhailovsky" <an...@arhont.com.INVALID>
> > To: "users" <us...@cloudstack.apache.org>, "users" <us...@cloudstack.apache.org>
> > Sent: Wednesday, 21 February, 2018 22:10:25
> > Subject: Re: VR routing issues in Advanced Mode
>
> > Dag,
> >
> >
> >
> >
> > Yeah, we have egress traffic enabled. We also use VPCs on some of the networks
> > and they are also effected by this issue along with None VPC networks.
> >
> >
> >
> >
> > Any thoughts?
> >
> >
> >
> >
> > Andrei Mikhailovsky
> >
> >
> >
> >
> >
> >
> >
> > From: Dag Sonstebo
> >
> >
> > Sent: Wednesday 21 February, 18:30
> >
> >
> > Subject: Re: VR routing issues in Advanced Mode
> >
> >
> > To: users@cloudstack.apache.org
> >
> >
> >
> >
> >
> >
> > Hi Andrei,
> >
> >
> >
> >
> > Understand. To get all the obvious things out the way – have you allowed egress
> > traffic on the two networks (you mention ACLs which we only use on VPCs and
> > basic networks)?
> >
> >
> >
> >
> > Regards,
> >
> >
> > Dag Sonstebo
> >
> >
> > Cloud Architect
> >
> >
> > ShapeBlue
> >
> >
> >
> >
> > On 21/02/2018, 14:51, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
> >
> >
> >
> >
> > Hi Dag,
> >
> >
> >
> >
> > Please see my comments below:
> >
> >
> >
> >
> >> Hi Andrei,
> >
> >
> >>
> >
> >
> >> You’re confusing the matter with your masking of public IP ranges. You said you
> >
> >
> >> have “2 x Public IP ranges with /26 netmask” – but since you are masking them
> >
> >
> >> out with X’s your email doesn’t make sense. If all the X’s are the same then a
> >
> >
> >> .10 and a .20 IP address would be on the same /26 network.
> >
> >
> >>
> >
> >
> >> I will assume that you do in fact have 2 x 26-bit networks, e.g.:
> >
> >
> >>
> >
> >
> >> 192.168.0.0/26 – with default gateway 192.168.0.1
> >
> >
> >> 192.168.0.64/26 – with default gateway 192.168.0.65
> >
> >
> >>
> >
> >
> >
> >
> >
> >
> > That is correct. I do have two separate /26 networks similar to what you've
> > described above. However, one /26 is used for direct public IP service
> > offering, where VRs are not involved in networking at all and the second /26 is
> > used for the service offering where VRs are used to provide the networking
> > function.
> >
> >
> >
> >
> >
> >
> >> If your two guest networks have VRs on separate public IP ranges you will have
> >
> >
> >> e.g.
> >
> >
> >>
> >
> >
> >> VR1: public IP 192.168.0.10
> >
> >
> >> VR2: public IP 192.168.0.70
> >
> >
> >>
> >
> >
> >
> >
> >
> >
> > Nope, the guest vms with VRs that can't talk to each other are on the same /26
> > network. (in your example that would be on the same 192.168.0.0/26)
> >
> >
> >
> >
> >
> >
> >> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up
> >
> >
> >> routing and possibly firewalling on the data centre device which handles the
> >
> >
> >> default gateway for the two networks – i.e. the top of rack switch or router
> >
> >
> >> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you
> >
> >
> >> can reach services on both networks from outside this range makes sense.
> >
> >
> >>
> >
> >
> >
> >
> > These has been set up and vms between separate /26 networks CAN talk to each
> > other. The VMs on the same /26 network that doesn't use the VR service can also
> > talk to each other. The problem with VMs on the same /26 that use VRs can't
> > talk to each other using their public IP addresses.
> >
> >
> >
> >
> >
> >
> >> So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER > VR2
> >
> >
> >> > VM2.
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> Regards,
> >
> >
> >> Dag Sonstebo
> >
> >
> >> Cloud Architect
> >
> >
> >> ShapeBlue
> >
> >
> >>
> >
> >
> >> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
> >
> >
> >>
> >
> >
> >> Hello
> >
> >
> >>
> >
> >
> >> Could someone help me to identify the routing issues that we have. The problem
> >
> >
> >> is the traffic from different guest networks can not reach each other via the
> >
> >
> >> public IPs.
> >
> >
> >>
> >
> >
> >> Here is my ACS setup:
> >
> >
> >> ACS 4.9.3.0 (both management and agents)
> >
> >
> >> KVM Hypervisor based on Ubuntu 16.04
> >
> >
> >> Ceph as primary storage. NFS as secondary storage
> >
> >
> >> Advanced Networking with vlan separation
> >
> >
> >> 2 x Public IP ranges with /26 netmask.
> >
> >
> >>
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> Here is an example when routing DOES NOT work:
> >
> >
> >>
> >
> >
> >> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide
> >
> >
> >> all networking services (dhcp, fw, port forwarding, load balancing, etc)
> >
> >
> >>
> >
> >
> >> Guest Network 1:
> >
> >
> >>
> >
> >
> >> Public IP: XXX.XXX.XXX.10/26
> >
> >
> >> Private IP range: 10.1.1.0/24
> >
> >
> >> guest vm1 IP: 10.1.1.100/24
> >
> >
> >>
> >
> >
> >> Guest Network 2:
> >
> >
> >> Public IP: XXX.XXX.XXX.20/26
> >
> >
> >> Private IP range: 10.1.1.0/24
> >
> >
> >> guest vm2 IP: 10.1.1.200/24
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port
> >
> >
> >> 80. I've created the port forwarding rules to forward port 80 from public
> >
> >
> >> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
> >
> >
> >> respectively.
> >
> >
> >>
> >
> >
> >> This setup works perfectly well when I am initiating the connections from
> >
> >
> >> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the
> >
> >
> >> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP
> >
> >
> >> XXX.XXX.XXX.20.
> >
> >
> >>
> >
> >
> >>
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> Here is an example when the routing DOES work:
> >
> >
> >>
> >
> >
> >> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are
> >
> >
> >> given directly to a guest vm
> >
> >
> >>
> >
> >
> >> Guest Network 1:
> >
> >
> >>
> >
> >
> >> guest vm1 Public IP: XXX.XXX.XXX.100/26
> >
> >
> >>
> >
> >
> >> Guest Network 2:
> >
> >
> >>
> >
> >
> >> guest vm2 Public IP: XXX.XXX.XXX.110/26
> >
> >
> >>
> >
> >
> >> In the Case 2, the guest vm has a public IP address directly assigned to its
> >
> >
> >> network interface. VRs are not used for this networking. Each guest has a fw
> >
> >
> >> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can
> >
> >
> >> access each other on port 80. Also, vms from Case 1 above can access port 80 on
> >
> >
> >> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case
> >
> >
> >> 1.
> >
> >
> >>
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> So, it seems that the rules on the VR in Case 1 do not allow traffic that
> >
> >
> >> originates from other VRs within the same public network range. The trace route
> >
> >
> >> shows the last hop being the VR's private IP address. How do I change that
> >
> >
> >> behaviour and fix the networking issue?
> >
> >
> >>
> >
> >
> >> Thanks
> >
> >
> >>
> >
> >
> >> Andrei
> >
> >
> >>
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> Dag.Sonstebo@shapeblue.com
> >
> >
> >> www.shapeblue.com
> >
> >
> >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> >
> >
> >> @shapeblue
> >
> >
> >
> >
> >
> >
> >
> >
> > Dag.Sonstebo@shapeblue.com
> >
> >
> > www.shapeblue.com
> >
> >
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> >
> >
> > @shapeblue
Re: VR routing issues in Advanced Mode
Posted by Andrei Mikhailovsky <an...@arhont.com.INVALID>.
Hi Dag,
Thanks, for your reply which I've missed earlier.
I have done some more digging around and would like to make some corrections to the problem at hand.
1. It seems that the problem only effects VPC networks within cloudstack.
2. Networks which use non-VPC networking can talk to each other.
3. VPC to VPC traffic is not working.
4. VPC traffic can reach non-VPC network
5. non-VPC traffic can't reach VPC network
In all cases, if the icmp is allowed in the ACLs, all networks can ping each other. So, the traffic is being routed and reaches the virtual router.
Any advice?
Thanks
----- Original Message -----
> From: "Dag Sonstebo" <Da...@shapeblue.com>
> To: "users" <us...@cloudstack.apache.org>
> Sent: Friday, 23 February, 2018 16:45:04
> Subject: Re: VR routing issues in Advanced Mode
> Hi Andrei,
>
> Next step is to do some tcpdumping on the two VRs. Set some ping’s going and
> check:
>
> Private NIC: tcpdump -i eth0 icmp
> Public NIC: tcpdump -i eth2 icmp
>
> That way you should be able to see how far your traffic reaches.
>
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 23/02/2018, 16:17, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
>
> Bump.
>
> Any ideas anyone? This issue is really annoying.
>
> Thanks
>
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
> ----- Original Message -----
> > From: "Andrei Mikhailovsky" <an...@arhont.com.INVALID>
> > To: "users" <us...@cloudstack.apache.org>, "users" <us...@cloudstack.apache.org>
> > Sent: Wednesday, 21 February, 2018 22:10:25
> > Subject: Re: VR routing issues in Advanced Mode
>
> > Dag,
> >
> >
> >
> >
> > Yeah, we have egress traffic enabled. We also use VPCs on some of the networks
> > and they are also effected by this issue along with None VPC networks.
> >
> >
> >
> >
> > Any thoughts?
> >
> >
> >
> >
> > Andrei Mikhailovsky
> >
> >
> >
> >
> >
> >
> >
> > From: Dag Sonstebo
> >
> >
> > Sent: Wednesday 21 February, 18:30
> >
> >
> > Subject: Re: VR routing issues in Advanced Mode
> >
> >
> > To: users@cloudstack.apache.org
> >
> >
> >
> >
> >
> >
> > Hi Andrei,
> >
> >
> >
> >
> > Understand. To get all the obvious things out the way – have you allowed egress
> > traffic on the two networks (you mention ACLs which we only use on VPCs and
> > basic networks)?
> >
> >
> >
> >
> > Regards,
> >
> >
> > Dag Sonstebo
> >
> >
> > Cloud Architect
> >
> >
> > ShapeBlue
> >
> >
> >
> >
> > On 21/02/2018, 14:51, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
> >
> >
> >
> >
> > Hi Dag,
> >
> >
> >
> >
> > Please see my comments below:
> >
> >
> >
> >
> >> Hi Andrei,
> >
> >
> >>
> >
> >
> >> You’re confusing the matter with your masking of public IP ranges. You said you
> >
> >
> >> have “2 x Public IP ranges with /26 netmask” – but since you are masking them
> >
> >
> >> out with X’s your email doesn’t make sense. If all the X’s are the same then a
> >
> >
> >> .10 and a .20 IP address would be on the same /26 network.
> >
> >
> >>
> >
> >
> >> I will assume that you do in fact have 2 x 26-bit networks, e.g.:
> >
> >
> >>
> >
> >
> >> 192.168.0.0/26 – with default gateway 192.168.0.1
> >
> >
> >> 192.168.0.64/26 – with default gateway 192.168.0.65
> >
> >
> >>
> >
> >
> >
> >
> >
> >
> > That is correct. I do have two separate /26 networks similar to what you've
> > described above. However, one /26 is used for direct public IP service
> > offering, where VRs are not involved in networking at all and the second /26 is
> > used for the service offering where VRs are used to provide the networking
> > function.
> >
> >
> >
> >
> >
> >
> >> If your two guest networks have VRs on separate public IP ranges you will have
> >
> >
> >> e.g.
> >
> >
> >>
> >
> >
> >> VR1: public IP 192.168.0.10
> >
> >
> >> VR2: public IP 192.168.0.70
> >
> >
> >>
> >
> >
> >
> >
> >
> >
> > Nope, the guest vms with VRs that can't talk to each other are on the same /26
> > network. (in your example that would be on the same 192.168.0.0/26)
> >
> >
> >
> >
> >
> >
> >> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up
> >
> >
> >> routing and possibly firewalling on the data centre device which handles the
> >
> >
> >> default gateway for the two networks – i.e. the top of rack switch or router
> >
> >
> >> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you
> >
> >
> >> can reach services on both networks from outside this range makes sense.
> >
> >
> >>
> >
> >
> >
> >
> > These has been set up and vms between separate /26 networks CAN talk to each
> > other. The VMs on the same /26 network that doesn't use the VR service can also
> > talk to each other. The problem with VMs on the same /26 that use VRs can't
> > talk to each other using their public IP addresses.
> >
> >
> >
> >
> >
> >
> >> So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER > VR2
> >
> >
> >> > VM2.
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> Regards,
> >
> >
> >> Dag Sonstebo
> >
> >
> >> Cloud Architect
> >
> >
> >> ShapeBlue
> >
> >
> >>
> >
> >
> >> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
> >
> >
> >>
> >
> >
> >> Hello
> >
> >
> >>
> >
> >
> >> Could someone help me to identify the routing issues that we have. The problem
> >
> >
> >> is the traffic from different guest networks can not reach each other via the
> >
> >
> >> public IPs.
> >
> >
> >>
> >
> >
> >> Here is my ACS setup:
> >
> >
> >> ACS 4.9.3.0 (both management and agents)
> >
> >
> >> KVM Hypervisor based on Ubuntu 16.04
> >
> >
> >> Ceph as primary storage. NFS as secondary storage
> >
> >
> >> Advanced Networking with vlan separation
> >
> >
> >> 2 x Public IP ranges with /26 netmask.
> >
> >
> >>
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> Here is an example when routing DOES NOT work:
> >
> >
> >>
> >
> >
> >> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide
> >
> >
> >> all networking services (dhcp, fw, port forwarding, load balancing, etc)
> >
> >
> >>
> >
> >
> >> Guest Network 1:
> >
> >
> >>
> >
> >
> >> Public IP: XXX.XXX.XXX.10/26
> >
> >
> >> Private IP range: 10.1.1.0/24
> >
> >
> >> guest vm1 IP: 10.1.1.100/24
> >
> >
> >>
> >
> >
> >> Guest Network 2:
> >
> >
> >> Public IP: XXX.XXX.XXX.20/26
> >
> >
> >> Private IP range: 10.1.1.0/24
> >
> >
> >> guest vm2 IP: 10.1.1.200/24
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port
> >
> >
> >> 80. I've created the port forwarding rules to forward port 80 from public
> >
> >
> >> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
> >
> >
> >> respectively.
> >
> >
> >>
> >
> >
> >> This setup works perfectly well when I am initiating the connections from
> >
> >
> >> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the
> >
> >
> >> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP
> >
> >
> >> XXX.XXX.XXX.20.
> >
> >
> >>
> >
> >
> >>
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> Here is an example when the routing DOES work:
> >
> >
> >>
> >
> >
> >> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are
> >
> >
> >> given directly to a guest vm
> >
> >
> >>
> >
> >
> >> Guest Network 1:
> >
> >
> >>
> >
> >
> >> guest vm1 Public IP: XXX.XXX.XXX.100/26
> >
> >
> >>
> >
> >
> >> Guest Network 2:
> >
> >
> >>
> >
> >
> >> guest vm2 Public IP: XXX.XXX.XXX.110/26
> >
> >
> >>
> >
> >
> >> In the Case 2, the guest vm has a public IP address directly assigned to its
> >
> >
> >> network interface. VRs are not used for this networking. Each guest has a fw
> >
> >
> >> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can
> >
> >
> >> access each other on port 80. Also, vms from Case 1 above can access port 80 on
> >
> >
> >> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case
> >
> >
> >> 1.
> >
> >
> >>
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> So, it seems that the rules on the VR in Case 1 do not allow traffic that
> >
> >
> >> originates from other VRs within the same public network range. The trace route
> >
> >
> >> shows the last hop being the VR's private IP address. How do I change that
> >
> >
> >> behaviour and fix the networking issue?
> >
> >
> >>
> >
> >
> >> Thanks
> >
> >
> >>
> >
> >
> >> Andrei
> >
> >
> >>
> >
> >
> >>
> >
> >
> >>
> >
> >
> >> Dag.Sonstebo@shapeblue.com
> >
> >
> >> www.shapeblue.com
> >
> >
> >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> >
> >
> >> @shapeblue
> >
> >
> >
> >
> >
> >
> >
> >
> > Dag.Sonstebo@shapeblue.com
> >
> >
> > www.shapeblue.com
> >
> >
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> >
> >
> > @shapeblue
Re: VR routing issues in Advanced Mode
Posted by Dag Sonstebo <Da...@shapeblue.com>.
Hi Andrei,
Next step is to do some tcpdumping on the two VRs. Set some ping’s going and check:
Private NIC: tcpdump -i eth0 icmp
Public NIC: tcpdump -i eth2 icmp
That way you should be able to see how far your traffic reaches.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 23/02/2018, 16:17, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
Bump.
Any ideas anyone? This issue is really annoying.
Thanks
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
----- Original Message -----
> From: "Andrei Mikhailovsky" <an...@arhont.com.INVALID>
> To: "users" <us...@cloudstack.apache.org>, "users" <us...@cloudstack.apache.org>
> Sent: Wednesday, 21 February, 2018 22:10:25
> Subject: Re: VR routing issues in Advanced Mode
> Dag,
>
>
>
>
> Yeah, we have egress traffic enabled. We also use VPCs on some of the networks
> and they are also effected by this issue along with None VPC networks.
>
>
>
>
> Any thoughts?
>
>
>
>
> Andrei Mikhailovsky
>
>
>
>
>
>
>
> From: Dag Sonstebo
>
>
> Sent: Wednesday 21 February, 18:30
>
>
> Subject: Re: VR routing issues in Advanced Mode
>
>
> To: users@cloudstack.apache.org
>
>
>
>
>
>
> Hi Andrei,
>
>
>
>
> Understand. To get all the obvious things out the way – have you allowed egress
> traffic on the two networks (you mention ACLs which we only use on VPCs and
> basic networks)?
>
>
>
>
> Regards,
>
>
> Dag Sonstebo
>
>
> Cloud Architect
>
>
> ShapeBlue
>
>
>
>
> On 21/02/2018, 14:51, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
>
>
>
>
> Hi Dag,
>
>
>
>
> Please see my comments below:
>
>
>
>
>> Hi Andrei,
>
>
>>
>
>
>> You’re confusing the matter with your masking of public IP ranges. You said you
>
>
>> have “2 x Public IP ranges with /26 netmask” – but since you are masking them
>
>
>> out with X’s your email doesn’t make sense. If all the X’s are the same then a
>
>
>> .10 and a .20 IP address would be on the same /26 network.
>
>
>>
>
>
>> I will assume that you do in fact have 2 x 26-bit networks, e.g.:
>
>
>>
>
>
>> 192.168.0.0/26 – with default gateway 192.168.0.1
>
>
>> 192.168.0.64/26 – with default gateway 192.168.0.65
>
>
>>
>
>
>
>
>
>
> That is correct. I do have two separate /26 networks similar to what you've
> described above. However, one /26 is used for direct public IP service
> offering, where VRs are not involved in networking at all and the second /26 is
> used for the service offering where VRs are used to provide the networking
> function.
>
>
>
>
>
>
>> If your two guest networks have VRs on separate public IP ranges you will have
>
>
>> e.g.
>
>
>>
>
>
>> VR1: public IP 192.168.0.10
>
>
>> VR2: public IP 192.168.0.70
>
>
>>
>
>
>
>
>
>
> Nope, the guest vms with VRs that can't talk to each other are on the same /26
> network. (in your example that would be on the same 192.168.0.0/26)
>
>
>
>
>
>
>> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up
>
>
>> routing and possibly firewalling on the data centre device which handles the
>
>
>> default gateway for the two networks – i.e. the top of rack switch or router
>
>
>> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you
>
>
>> can reach services on both networks from outside this range makes sense.
>
>
>>
>
>
>
>
> These has been set up and vms between separate /26 networks CAN talk to each
> other. The VMs on the same /26 network that doesn't use the VR service can also
> talk to each other. The problem with VMs on the same /26 that use VRs can't
> talk to each other using their public IP addresses.
>
>
>
>
>
>
>> So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER > VR2
>
>
>> > VM2.
>
>
>>
>
>
>>
>
>
>> Regards,
>
>
>> Dag Sonstebo
>
>
>> Cloud Architect
>
>
>> ShapeBlue
>
>
>>
>
>
>> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
>
>
>>
>
>
>> Hello
>
>
>>
>
>
>> Could someone help me to identify the routing issues that we have. The problem
>
>
>> is the traffic from different guest networks can not reach each other via the
>
>
>> public IPs.
>
>
>>
>
>
>> Here is my ACS setup:
>
>
>> ACS 4.9.3.0 (both management and agents)
>
>
>> KVM Hypervisor based on Ubuntu 16.04
>
>
>> Ceph as primary storage. NFS as secondary storage
>
>
>> Advanced Networking with vlan separation
>
>
>> 2 x Public IP ranges with /26 netmask.
>
>
>>
>
>
>>
>
>
>>
>
>
>> Here is an example when routing DOES NOT work:
>
>
>>
>
>
>> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide
>
>
>> all networking services (dhcp, fw, port forwarding, load balancing, etc)
>
>
>>
>
>
>> Guest Network 1:
>
>
>>
>
>
>> Public IP: XXX.XXX.XXX.10/26
>
>
>> Private IP range: 10.1.1.0/24
>
>
>> guest vm1 IP: 10.1.1.100/24
>
>
>>
>
>
>> Guest Network 2:
>
>
>> Public IP: XXX.XXX.XXX.20/26
>
>
>> Private IP range: 10.1.1.0/24
>
>
>> guest vm2 IP: 10.1.1.200/24
>
>
>>
>
>
>>
>
>
>> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port
>
>
>> 80. I've created the port forwarding rules to forward port 80 from public
>
>
>> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
>
>
>> respectively.
>
>
>>
>
>
>> This setup works perfectly well when I am initiating the connections from
>
>
>> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the
>
>
>> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP
>
>
>> XXX.XXX.XXX.20.
>
>
>>
>
>
>>
>
>
>>
>
>
>>
>
>
>> Here is an example when the routing DOES work:
>
>
>>
>
>
>> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are
>
>
>> given directly to a guest vm
>
>
>>
>
>
>> Guest Network 1:
>
>
>>
>
>
>> guest vm1 Public IP: XXX.XXX.XXX.100/26
>
>
>>
>
>
>> Guest Network 2:
>
>
>>
>
>
>> guest vm2 Public IP: XXX.XXX.XXX.110/26
>
>
>>
>
>
>> In the Case 2, the guest vm has a public IP address directly assigned to its
>
>
>> network interface. VRs are not used for this networking. Each guest has a fw
>
>
>> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can
>
>
>> access each other on port 80. Also, vms from Case 1 above can access port 80 on
>
>
>> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case
>
>
>> 1.
>
>
>>
>
>
>>
>
>
>>
>
>
>> So, it seems that the rules on the VR in Case 1 do not allow traffic that
>
>
>> originates from other VRs within the same public network range. The trace route
>
>
>> shows the last hop being the VR's private IP address. How do I change that
>
>
>> behaviour and fix the networking issue?
>
>
>>
>
>
>> Thanks
>
>
>>
>
>
>> Andrei
>
>
>>
>
>
>>
>
>
>>
>
>
>> Dag.Sonstebo@shapeblue.com
>
>
>> www.shapeblue.com
>
>
>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>
>
>> @shapeblue
>
>
>
>
>
>
>
>
> Dag.Sonstebo@shapeblue.com
>
>
> www.shapeblue.com
>
>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>
>
> @shapeblue
Re: VR routing issues in Advanced Mode
Posted by Andrei Mikhailovsky <an...@arhont.com.INVALID>.
Bump.
Any ideas anyone? This issue is really annoying.
Thanks
----- Original Message -----
> From: "Andrei Mikhailovsky" <an...@arhont.com.INVALID>
> To: "users" <us...@cloudstack.apache.org>, "users" <us...@cloudstack.apache.org>
> Sent: Wednesday, 21 February, 2018 22:10:25
> Subject: Re: VR routing issues in Advanced Mode
> Dag,
>
>
>
>
> Yeah, we have egress traffic enabled. We also use VPCs on some of the networks
> and they are also effected by this issue along with None VPC networks.
>
>
>
>
> Any thoughts?
>
>
>
>
> Andrei Mikhailovsky
>
>
>
>
>
>
>
> From: Dag Sonstebo
>
>
> Sent: Wednesday 21 February, 18:30
>
>
> Subject: Re: VR routing issues in Advanced Mode
>
>
> To: users@cloudstack.apache.org
>
>
>
>
>
>
> Hi Andrei,
>
>
>
>
> Understand. To get all the obvious things out the way – have you allowed egress
> traffic on the two networks (you mention ACLs which we only use on VPCs and
> basic networks)?
>
>
>
>
> Regards,
>
>
> Dag Sonstebo
>
>
> Cloud Architect
>
>
> ShapeBlue
>
>
>
>
> On 21/02/2018, 14:51, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
>
>
>
>
> Hi Dag,
>
>
>
>
> Please see my comments below:
>
>
>
>
>> Hi Andrei,
>
>
>>
>
>
>> You’re confusing the matter with your masking of public IP ranges. You said you
>
>
>> have “2 x Public IP ranges with /26 netmask” – but since you are masking them
>
>
>> out with X’s your email doesn’t make sense. If all the X’s are the same then a
>
>
>> .10 and a .20 IP address would be on the same /26 network.
>
>
>>
>
>
>> I will assume that you do in fact have 2 x 26-bit networks, e.g.:
>
>
>>
>
>
>> 192.168.0.0/26 – with default gateway 192.168.0.1
>
>
>> 192.168.0.64/26 – with default gateway 192.168.0.65
>
>
>>
>
>
>
>
>
>
> That is correct. I do have two separate /26 networks similar to what you've
> described above. However, one /26 is used for direct public IP service
> offering, where VRs are not involved in networking at all and the second /26 is
> used for the service offering where VRs are used to provide the networking
> function.
>
>
>
>
>
>
>> If your two guest networks have VRs on separate public IP ranges you will have
>
>
>> e.g.
>
>
>>
>
>
>> VR1: public IP 192.168.0.10
>
>
>> VR2: public IP 192.168.0.70
>
>
>>
>
>
>
>
>
>
> Nope, the guest vms with VRs that can't talk to each other are on the same /26
> network. (in your example that would be on the same 192.168.0.0/26)
>
>
>
>
>
>
>> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up
>
>
>> routing and possibly firewalling on the data centre device which handles the
>
>
>> default gateway for the two networks – i.e. the top of rack switch or router
>
>
>> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you
>
>
>> can reach services on both networks from outside this range makes sense.
>
>
>>
>
>
>
>
> These has been set up and vms between separate /26 networks CAN talk to each
> other. The VMs on the same /26 network that doesn't use the VR service can also
> talk to each other. The problem with VMs on the same /26 that use VRs can't
> talk to each other using their public IP addresses.
>
>
>
>
>
>
>> So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER > VR2
>
>
>> > VM2.
>
>
>>
>
>
>>
>
>
>> Regards,
>
>
>> Dag Sonstebo
>
>
>> Cloud Architect
>
>
>> ShapeBlue
>
>
>>
>
>
>> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
>
>
>>
>
>
>> Hello
>
>
>>
>
>
>> Could someone help me to identify the routing issues that we have. The problem
>
>
>> is the traffic from different guest networks can not reach each other via the
>
>
>> public IPs.
>
>
>>
>
>
>> Here is my ACS setup:
>
>
>> ACS 4.9.3.0 (both management and agents)
>
>
>> KVM Hypervisor based on Ubuntu 16.04
>
>
>> Ceph as primary storage. NFS as secondary storage
>
>
>> Advanced Networking with vlan separation
>
>
>> 2 x Public IP ranges with /26 netmask.
>
>
>>
>
>
>>
>
>
>>
>
>
>> Here is an example when routing DOES NOT work:
>
>
>>
>
>
>> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide
>
>
>> all networking services (dhcp, fw, port forwarding, load balancing, etc)
>
>
>>
>
>
>> Guest Network 1:
>
>
>>
>
>
>> Public IP: XXX.XXX.XXX.10/26
>
>
>> Private IP range: 10.1.1.0/24
>
>
>> guest vm1 IP: 10.1.1.100/24
>
>
>>
>
>
>> Guest Network 2:
>
>
>> Public IP: XXX.XXX.XXX.20/26
>
>
>> Private IP range: 10.1.1.0/24
>
>
>> guest vm2 IP: 10.1.1.200/24
>
>
>>
>
>
>>
>
>
>> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port
>
>
>> 80. I've created the port forwarding rules to forward port 80 from public
>
>
>> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
>
>
>> respectively.
>
>
>>
>
>
>> This setup works perfectly well when I am initiating the connections from
>
>
>> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the
>
>
>> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP
>
>
>> XXX.XXX.XXX.20.
>
>
>>
>
>
>>
>
>
>>
>
>
>>
>
>
>> Here is an example when the routing DOES work:
>
>
>>
>
>
>> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are
>
>
>> given directly to a guest vm
>
>
>>
>
>
>> Guest Network 1:
>
>
>>
>
>
>> guest vm1 Public IP: XXX.XXX.XXX.100/26
>
>
>>
>
>
>> Guest Network 2:
>
>
>>
>
>
>> guest vm2 Public IP: XXX.XXX.XXX.110/26
>
>
>>
>
>
>> In the Case 2, the guest vm has a public IP address directly assigned to its
>
>
>> network interface. VRs are not used for this networking. Each guest has a fw
>
>
>> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can
>
>
>> access each other on port 80. Also, vms from Case 1 above can access port 80 on
>
>
>> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case
>
>
>> 1.
>
>
>>
>
>
>>
>
>
>>
>
>
>> So, it seems that the rules on the VR in Case 1 do not allow traffic that
>
>
>> originates from other VRs within the same public network range. The trace route
>
>
>> shows the last hop being the VR's private IP address. How do I change that
>
>
>> behaviour and fix the networking issue?
>
>
>>
>
>
>> Thanks
>
>
>>
>
>
>> Andrei
>
>
>>
>
>
>>
>
>
>>
>
>
>> Dag.Sonstebo@shapeblue.com
>
>
>> www.shapeblue.com
>
>
>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>
>
>> @shapeblue
>
>
>
>
>
>
>
>
> Dag.Sonstebo@shapeblue.com
>
>
> www.shapeblue.com
>
>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>
>
> @shapeblue
Re: VR routing issues in Advanced Mode
Posted by Andrei Mikhailovsky <an...@arhont.com.INVALID>.
Dag,
Yeah, we have egress traffic enabled. We also use VPCs on some of the networks and they are also effected by this issue along with None VPC networks.
Any thoughts?
Andrei Mikhailovsky
From: Dag Sonstebo
Sent: Wednesday 21 February, 18:30
Subject: Re: VR routing issues in Advanced Mode
To: users@cloudstack.apache.org
Hi Andrei,
Understand. To get all the obvious things out the way – have you allowed egress traffic on the two networks (you mention ACLs which we only use on VPCs and basic networks)?
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 21/02/2018, 14:51, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
Hi Dag,
Please see my comments below:
> Hi Andrei,
>
> You’re confusing the matter with your masking of public IP ranges. You said you
> have “2 x Public IP ranges with /26 netmask” – but since you are masking them
> out with X’s your email doesn’t make sense. If all the X’s are the same then a
> .10 and a .20 IP address would be on the same /26 network.
>
> I will assume that you do in fact have 2 x 26-bit networks, e.g.:
>
> 192.168.0.0/26 – with default gateway 192.168.0.1
> 192.168.0.64/26 – with default gateway 192.168.0.65
>
That is correct. I do have two separate /26 networks similar to what you've described above. However, one /26 is used for direct public IP service offering, where VRs are not involved in networking at all and the second /26 is used for the service offering where VRs are used to provide the networking function.
> If your two guest networks have VRs on separate public IP ranges you will have
> e.g.
>
> VR1: public IP 192.168.0.10
> VR2: public IP 192.168.0.70
>
Nope, the guest vms with VRs that can't talk to each other are on the same /26 network. (in your example that would be on the same 192.168.0.0/26)
> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up
> routing and possibly firewalling on the data centre device which handles the
> default gateway for the two networks – i.e. the top of rack switch or router
> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you
> can reach services on both networks from outside this range makes sense.
>
These has been set up and vms between separate /26 networks CAN talk to each other. The VMs on the same /26 network that doesn't use the VR service can also talk to each other. The problem with VMs on the same /26 that use VRs can't talk to each other using their public IP addresses.
> So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER > VR2
> > VM2.
>
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
>
> Hello
>
> Could someone help me to identify the routing issues that we have. The problem
> is the traffic from different guest networks can not reach each other via the
> public IPs.
>
> Here is my ACS setup:
> ACS 4.9.3.0 (both management and agents)
> KVM Hypervisor based on Ubuntu 16.04
> Ceph as primary storage. NFS as secondary storage
> Advanced Networking with vlan separation
> 2 x Public IP ranges with /26 netmask.
>
>
>
> Here is an example when routing DOES NOT work:
>
> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide
> all networking services (dhcp, fw, port forwarding, load balancing, etc)
>
> Guest Network 1:
>
> Public IP: XXX.XXX.XXX.10/26
> Private IP range: 10.1.1.0/24
> guest vm1 IP: 10.1.1.100/24
>
> Guest Network 2:
> Public IP: XXX.XXX.XXX.20/26
> Private IP range: 10.1.1.0/24
> guest vm2 IP: 10.1.1.200/24
>
>
> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port
> 80. I've created the port forwarding rules to forward port 80 from public
> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
> respectively.
>
> This setup works perfectly well when I am initiating the connections from
> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the
> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP
> XXX.XXX.XXX.20.
>
>
>
>
> Here is an example when the routing DOES work:
>
> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are
> given directly to a guest vm
>
> Guest Network 1:
>
> guest vm1 Public IP: XXX.XXX.XXX.100/26
>
> Guest Network 2:
>
> guest vm2 Public IP: XXX.XXX.XXX.110/26
>
> In the Case 2, the guest vm has a public IP address directly assigned to its
> network interface. VRs are not used for this networking. Each guest has a fw
> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can
> access each other on port 80. Also, vms from Case 1 above can access port 80 on
> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case
> 1.
>
>
>
> So, it seems that the rules on the VR in Case 1 do not allow traffic that
> originates from other VRs within the same public network range. The trace route
> shows the last hop being the VR's private IP address. How do I change that
> behaviour and fix the networking issue?
>
> Thanks
>
> Andrei
>
>
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: VR routing issues in Advanced Mode
Posted by Dag Sonstebo <Da...@shapeblue.com>.
Hi Andrei,
Understand. To get all the obvious things out the way – have you allowed egress traffic on the two networks (you mention ACLs which we only use on VPCs and basic networks)?
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 21/02/2018, 14:51, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
Hi Dag,
Please see my comments below:
> Hi Andrei,
>
> You’re confusing the matter with your masking of public IP ranges. You said you
> have “2 x Public IP ranges with /26 netmask” – but since you are masking them
> out with X’s your email doesn’t make sense. If all the X’s are the same then a
> .10 and a .20 IP address would be on the same /26 network.
>
> I will assume that you do in fact have 2 x 26-bit networks, e.g.:
>
> 192.168.0.0/26 – with default gateway 192.168.0.1
> 192.168.0.64/26 – with default gateway 192.168.0.65
>
That is correct. I do have two separate /26 networks similar to what you've described above. However, one /26 is used for direct public IP service offering, where VRs are not involved in networking at all and the second /26 is used for the service offering where VRs are used to provide the networking function.
> If your two guest networks have VRs on separate public IP ranges you will have
> e.g.
>
> VR1: public IP 192.168.0.10
> VR2: public IP 192.168.0.70
>
Nope, the guest vms with VRs that can't talk to each other are on the same /26 network. (in your example that would be on the same 192.168.0.0/26)
> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up
> routing and possibly firewalling on the data centre device which handles the
> default gateway for the two networks – i.e. the top of rack switch or router
> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you
> can reach services on both networks from outside this range makes sense.
>
These has been set up and vms between separate /26 networks CAN talk to each other. The VMs on the same /26 network that doesn't use the VR service can also talk to each other. The problem with VMs on the same /26 that use VRs can't talk to each other using their public IP addresses.
> So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER > VR2
> > VM2.
>
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
>
> Hello
>
> Could someone help me to identify the routing issues that we have. The problem
> is the traffic from different guest networks can not reach each other via the
> public IPs.
>
> Here is my ACS setup:
> ACS 4.9.3.0 (both management and agents)
> KVM Hypervisor based on Ubuntu 16.04
> Ceph as primary storage. NFS as secondary storage
> Advanced Networking with vlan separation
> 2 x Public IP ranges with /26 netmask.
>
>
>
> Here is an example when routing DOES NOT work:
>
> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide
> all networking services (dhcp, fw, port forwarding, load balancing, etc)
>
> Guest Network 1:
>
> Public IP: XXX.XXX.XXX.10/26
> Private IP range: 10.1.1.0/24
> guest vm1 IP: 10.1.1.100/24
>
> Guest Network 2:
> Public IP: XXX.XXX.XXX.20/26
> Private IP range: 10.1.1.0/24
> guest vm2 IP: 10.1.1.200/24
>
>
> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port
> 80. I've created the port forwarding rules to forward port 80 from public
> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
> respectively.
>
> This setup works perfectly well when I am initiating the connections from
> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the
> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP
> XXX.XXX.XXX.20.
>
>
>
>
> Here is an example when the routing DOES work:
>
> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are
> given directly to a guest vm
>
> Guest Network 1:
>
> guest vm1 Public IP: XXX.XXX.XXX.100/26
>
> Guest Network 2:
>
> guest vm2 Public IP: XXX.XXX.XXX.110/26
>
> In the Case 2, the guest vm has a public IP address directly assigned to its
> network interface. VRs are not used for this networking. Each guest has a fw
> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can
> access each other on port 80. Also, vms from Case 1 above can access port 80 on
> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case
> 1.
>
>
>
> So, it seems that the rules on the VR in Case 1 do not allow traffic that
> originates from other VRs within the same public network range. The trace route
> shows the last hop being the VR's private IP address. How do I change that
> behaviour and fix the networking issue?
>
> Thanks
>
> Andrei
>
>
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: VR routing issues in Advanced Mode
Posted by Andrei Mikhailovsky <an...@arhont.com.INVALID>.
Hi Dag,
Please see my comments below:
> Hi Andrei,
>
> You’re confusing the matter with your masking of public IP ranges. You said you
> have “2 x Public IP ranges with /26 netmask” – but since you are masking them
> out with X’s your email doesn’t make sense. If all the X’s are the same then a
> .10 and a .20 IP address would be on the same /26 network.
>
> I will assume that you do in fact have 2 x 26-bit networks, e.g.:
>
> 192.168.0.0/26 – with default gateway 192.168.0.1
> 192.168.0.64/26 – with default gateway 192.168.0.65
>
That is correct. I do have two separate /26 networks similar to what you've described above. However, one /26 is used for direct public IP service offering, where VRs are not involved in networking at all and the second /26 is used for the service offering where VRs are used to provide the networking function.
> If your two guest networks have VRs on separate public IP ranges you will have
> e.g.
>
> VR1: public IP 192.168.0.10
> VR2: public IP 192.168.0.70
>
Nope, the guest vms with VRs that can't talk to each other are on the same /26 network. (in your example that would be on the same 192.168.0.0/26)
> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up
> routing and possibly firewalling on the data centre device which handles the
> default gateway for the two networks – i.e. the top of rack switch or router
> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you
> can reach services on both networks from outside this range makes sense.
>
These has been set up and vms between separate /26 networks CAN talk to each other. The VMs on the same /26 network that doesn't use the VR service can also talk to each other. The problem with VMs on the same /26 that use VRs can't talk to each other using their public IP addresses.
> So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER > VR2
> > VM2.
>
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
>
> Hello
>
> Could someone help me to identify the routing issues that we have. The problem
> is the traffic from different guest networks can not reach each other via the
> public IPs.
>
> Here is my ACS setup:
> ACS 4.9.3.0 (both management and agents)
> KVM Hypervisor based on Ubuntu 16.04
> Ceph as primary storage. NFS as secondary storage
> Advanced Networking with vlan separation
> 2 x Public IP ranges with /26 netmask.
>
>
>
> Here is an example when routing DOES NOT work:
>
> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide
> all networking services (dhcp, fw, port forwarding, load balancing, etc)
>
> Guest Network 1:
>
> Public IP: XXX.XXX.XXX.10/26
> Private IP range: 10.1.1.0/24
> guest vm1 IP: 10.1.1.100/24
>
> Guest Network 2:
> Public IP: XXX.XXX.XXX.20/26
> Private IP range: 10.1.1.0/24
> guest vm2 IP: 10.1.1.200/24
>
>
> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port
> 80. I've created the port forwarding rules to forward port 80 from public
> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
> respectively.
>
> This setup works perfectly well when I am initiating the connections from
> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the
> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP
> XXX.XXX.XXX.20.
>
>
>
>
> Here is an example when the routing DOES work:
>
> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are
> given directly to a guest vm
>
> Guest Network 1:
>
> guest vm1 Public IP: XXX.XXX.XXX.100/26
>
> Guest Network 2:
>
> guest vm2 Public IP: XXX.XXX.XXX.110/26
>
> In the Case 2, the guest vm has a public IP address directly assigned to its
> network interface. VRs are not used for this networking. Each guest has a fw
> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can
> access each other on port 80. Also, vms from Case 1 above can access port 80 on
> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case
> 1.
>
>
>
> So, it seems that the rules on the VR in Case 1 do not allow traffic that
> originates from other VRs within the same public network range. The trace route
> shows the last hop being the VR's private IP address. How do I change that
> behaviour and fix the networking issue?
>
> Thanks
>
> Andrei
>
>
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
Re: VR routing issues in Advanced Mode
Posted by Dag Sonstebo <Da...@shapeblue.com>.
Hi Andrei,
You’re confusing the matter with your masking of public IP ranges. You said you have “2 x Public IP ranges with /26 netmask” – but since you are masking them out with X’s your email doesn’t make sense. If all the X’s are the same then a .10 and a .20 IP address would be on the same /26 network.
I will assume that you do in fact have 2 x 26-bit networks, e.g.:
192.168.0.0/26 – with default gateway 192.168.0.1
192.168.0.64/26 – with default gateway 192.168.0.65
If your two guest networks have VRs on separate public IP ranges you will have e.g.
VR1: public IP 192.168.0.10
VR2: public IP 192.168.0.70
For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up routing and possibly firewalling on the data centre device which handles the default gateway for the two networks – i.e. the top of rack switch or router which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you can reach services on both networks from outside this range makes sense.
So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER > VR2 > VM2.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 21/02/2018, 12:27, "Andrei Mikhailovsky" <an...@arhont.com.INVALID> wrote:
Hello
Could someone help me to identify the routing issues that we have. The problem is the traffic from different guest networks can not reach each other via the public IPs.
Here is my ACS setup:
ACS 4.9.3.0 (both management and agents)
KVM Hypervisor based on Ubuntu 16.04
Ceph as primary storage. NFS as secondary storage
Advanced Networking with vlan separation
2 x Public IP ranges with /26 netmask.
Here is an example when routing DOES NOT work:
Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide all networking services (dhcp, fw, port forwarding, load balancing, etc)
Guest Network 1:
Public IP: XXX.XXX.XXX.10/26
Private IP range: 10.1.1.0/24
guest vm1 IP: 10.1.1.100/24
Guest Network 2:
Public IP: XXX.XXX.XXX.20/26
Private IP range: 10.1.1.0/24
guest vm2 IP: 10.1.1.200/24
I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port 80. I've created the port forwarding rules to forward port 80 from public XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200 respectively.
This setup works perfectly well when I am initiating the connections from outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP XXX.XXX.XXX.20.
Here is an example when the routing DOES work:
Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are given directly to a guest vm
Guest Network 1:
guest vm1 Public IP: XXX.XXX.XXX.100/26
Guest Network 2:
guest vm2 Public IP: XXX.XXX.XXX.110/26
In the Case 2, the guest vm has a public IP address directly assigned to its network interface. VRs are not used for this networking. Each guest has a fw rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can access each other on port 80. Also, vms from Case 1 above can access port 80 on vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case 1.
So, it seems that the rules on the VR in Case 1 do not allow traffic that originates from other VRs within the same public network range. The trace route shows the last hop being the VR's private IP address. How do I change that behaviour and fix the networking issue?
Thanks
Andrei
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: VR routing issues in Advanced Mode
Posted by Andrei Mikhailovsky <an...@arhont.com.INVALID>.
Andrija,
the vms are trying to reach each other using the public IP addresses, not the private addresses.
Cheers
Andrei
----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: "users" <us...@cloudstack.apache.org>
> Sent: Wednesday, 21 February, 2018 12:48:57
> Subject: Re: VR routing issues in Advanced Mode
> Hi Andrei,
>
> you dont have typo in your input, right ?
>
> if I read this correctly, the case that don't work for you is as following:
>
> VR1 ( XXX.XXX.XXX.10/26) --> Guest1 Network / VM 10.1.1.100/24
>
> VR2 ( XXX.XXX.XXX.20/26)-- Guest1 Network / VM 10.1.1.200/24
>
> Is this correct ?
>
> If so, it's normal that VM1 can reach VM2 via following path VM1-->VR1 --->
> VR2 --> VM2:80 because both VM1 and VM2 are on the "same" subnet (
> 10.1.1.0/24) so the VM1 decides to BROADCAST traffic over "switch" to reach
> IP in the same network (VM2 IP 10.1.1.0). If this IP would be in the i.e.
> 10.2.1.0 netowrk, then VM would decided to send packet to it's default gtw
> (VR) and than things would work fine.
>
> Otherwise, if this is single VR, you actually can not even create 2
> networks with same subnet since both are (per your input, if not typo)
> 10.1.1.0/24 subnets
>
> ?
>
> Cheers
> Andrija
>
> On 21 February 2018 at 13:27, Andrei Mikhailovsky <andrei@arhont.com.invalid
>> wrote:
>
>> Hello
>>
>> Could someone help me to identify the routing issues that we have. The
>> problem is the traffic from different guest networks can not reach each
>> other via the public IPs.
>>
>> Here is my ACS setup:
>> ACS 4.9.3.0 (both management and agents)
>> KVM Hypervisor based on Ubuntu 16.04
>> Ceph as primary storage. NFS as secondary storage
>> Advanced Networking with vlan separation
>> 2 x Public IP ranges with /26 netmask.
>>
>>
>>
>> Here is an example when routing DOES NOT work:
>>
>> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and
>> provide all networking services (dhcp, fw, port forwarding, load balancing,
>> etc)
>>
>> Guest Network 1:
>>
>> Public IP: XXX.XXX.XXX.10/26
>> Private IP range: 10.1.1.0/24
>> guest vm1 IP: 10.1.1.100/24
>>
>> Guest Network 2:
>> Public IP: XXX.XXX.XXX.20/26
>> Private IP range: 10.1.1.0/24
>> guest vm2 IP: 10.1.1.200/24
>>
>>
>> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0
>> on port 80. I've created the port forwarding rules to forward port 80 from
>> public XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
>> respectively.
>>
>> This setup works perfectly well when I am initiating the connections from
>> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using
>> the public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public
>> IP XXX.XXX.XXX.20.
>>
>>
>>
>>
>> Here is an example when the routing DOES work:
>>
>> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public
>> IPs are given directly to a guest vm
>>
>> Guest Network 1:
>>
>> guest vm1 Public IP: XXX.XXX.XXX.100/26
>>
>> Guest Network 2:
>>
>> guest vm2 Public IP: XXX.XXX.XXX.110/26
>>
>> In the Case 2, the guest vm has a public IP address directly assigned to
>> its network interface. VRs are not used for this networking. Each guest has
>> a fw rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1
>> and vm2 can access each other on port 80. Also, vms from Case 1 above can
>> access port 80 on vms from Case 2, similarly, vms from Case 2 can access
>> port 80 on vms from Case 1.
>>
>>
>>
>> So, it seems that the rules on the VR in Case 1 do not allow traffic that
>> originates from other VRs within the same public network range. The trace
>> route shows the last hop being the VR's private IP address. How do I change
>> that behaviour and fix the networking issue?
>>
>> Thanks
>>
>> Andrei
>>
>
>
>
> --
>
> Andrija Panić
Re: VR routing issues in Advanced Mode
Posted by Andrija Panic <an...@gmail.com>.
Hi Andrei,
you dont have typo in your input, right ?
if I read this correctly, the case that don't work for you is as following:
VR1 ( XXX.XXX.XXX.10/26) --> Guest1 Network / VM 10.1.1.100/24
VR2 ( XXX.XXX.XXX.20/26)-- Guest1 Network / VM 10.1.1.200/24
Is this correct ?
If so, it's normal that VM1 can reach VM2 via following path VM1-->VR1 --->
VR2 --> VM2:80 because both VM1 and VM2 are on the "same" subnet (
10.1.1.0/24) so the VM1 decides to BROADCAST traffic over "switch" to reach
IP in the same network (VM2 IP 10.1.1.0). If this IP would be in the i.e.
10.2.1.0 netowrk, then VM would decided to send packet to it's default gtw
(VR) and than things would work fine.
Otherwise, if this is single VR, you actually can not even create 2
networks with same subnet since both are (per your input, if not typo)
10.1.1.0/24 subnets
?
Cheers
Andrija
On 21 February 2018 at 13:27, Andrei Mikhailovsky <andrei@arhont.com.invalid
> wrote:
> Hello
>
> Could someone help me to identify the routing issues that we have. The
> problem is the traffic from different guest networks can not reach each
> other via the public IPs.
>
> Here is my ACS setup:
> ACS 4.9.3.0 (both management and agents)
> KVM Hypervisor based on Ubuntu 16.04
> Ceph as primary storage. NFS as secondary storage
> Advanced Networking with vlan separation
> 2 x Public IP ranges with /26 netmask.
>
>
>
> Here is an example when routing DOES NOT work:
>
> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and
> provide all networking services (dhcp, fw, port forwarding, load balancing,
> etc)
>
> Guest Network 1:
>
> Public IP: XXX.XXX.XXX.10/26
> Private IP range: 10.1.1.0/24
> guest vm1 IP: 10.1.1.100/24
>
> Guest Network 2:
> Public IP: XXX.XXX.XXX.20/26
> Private IP range: 10.1.1.0/24
> guest vm2 IP: 10.1.1.200/24
>
>
> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0
> on port 80. I've created the port forwarding rules to forward port 80 from
> public XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
> respectively.
>
> This setup works perfectly well when I am initiating the connections from
> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using
> the public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public
> IP XXX.XXX.XXX.20.
>
>
>
>
> Here is an example when the routing DOES work:
>
> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public
> IPs are given directly to a guest vm
>
> Guest Network 1:
>
> guest vm1 Public IP: XXX.XXX.XXX.100/26
>
> Guest Network 2:
>
> guest vm2 Public IP: XXX.XXX.XXX.110/26
>
> In the Case 2, the guest vm has a public IP address directly assigned to
> its network interface. VRs are not used for this networking. Each guest has
> a fw rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1
> and vm2 can access each other on port 80. Also, vms from Case 1 above can
> access port 80 on vms from Case 2, similarly, vms from Case 2 can access
> port 80 on vms from Case 1.
>
>
>
> So, it seems that the rules on the VR in Case 1 do not allow traffic that
> originates from other VRs within the same public network range. The trace
> route shows the last hop being the VR's private IP address. How do I change
> that behaviour and fix the networking issue?
>
> Thanks
>
> Andrei
>
--
Andrija Panić