You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by "Musayev, Ilya" <im...@webmd.net> on 2012/12/03 19:59:09 UTC
Router VM and Network Issue
So I hit a glitch where a router VM boots up but does not really pass any traffic unless I ping the gateway of the CS host from within the router VM.
Once the gateway ping goes through, CS is able to SSH into a router VM and everything is fine and dandy..
But this behavior really puzzles me. Linux network stack is not fully activated or routing is not fully functional until the initial CS GW ping.
Also I cant ping/ssh the router VM from CS unless a initiate a ping from within the router VM.
I'm on CS 4.0 and vSphere5. This seem to affect the Advanced Network setup more than Basic because of routing complexity - as you add some routes into linux routing table.
Has anyone seen this before?
Re: Router VM and Network Issue
Posted by "Kelceydamage@bbits" <ke...@bbits.ca>.
I had a similar issue with KVM and CS on 3.0.2. Turns out the Debian squeeze routers/systemVMs were incompatible with the vIP HA method I was using upstream(ARP based HA-IPs). It's worth running tests against any physical or custom virtual routing equipment you might have in the network path.
Sent from my iPhone
On Dec 3, 2012, at 11:44 AM, "Musayev, Ilya" <im...@webmd.net> wrote:
> Marcus,
>
> I agree with you, except we don't drop any broadcast packets for sure on ESX. That would be very counter productive.
>
> Still puzzled - but you could be right - it has something to do with ARP.
>
> Regards
> ilya
>
> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: Monday, December 03, 2012 2:12 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: Router VM and Network Issue
>
> I don't have a vsphere environment, but it sounds a bit like an ARP issue, where perhaps the bridge on the VMware host is filtering broadcast or something. Maybe a workaround could be added to the system VM to make sure things are more robust, but it sounds like something in your network is disabling the discovery of your router VM via broadcast, and when you ping from the system vm then the MAC table on the switch is finally updated and things start working.
> On Dec 3, 2012 11:59 AM, "Musayev, Ilya" <im...@webmd.net> wrote:
>
>> So I hit a glitch where a router VM boots up but does not really pass
>> any traffic unless I ping the gateway of the CS host from within the router VM.
>>
>> Once the gateway ping goes through, CS is able to SSH into a router VM
>> and everything is fine and dandy..
>>
>> But this behavior really puzzles me. Linux network stack is not fully
>> activated or routing is not fully functional until the initial CS GW ping.
>>
>> Also I cant ping/ssh the router VM from CS unless a initiate a ping
>> from within the router VM.
>>
>> I'm on CS 4.0 and vSphere5. This seem to affect the Advanced Network
>> setup more than Basic because of routing complexity - as you add some
>> routes into linux routing table.
>>
>>
>> Has anyone seen this before?
>>
>
RE: Router VM and Network Issue
Posted by "Musayev, Ilya" <im...@webmd.net>.
Marcus,
I agree with you, except we don't drop any broadcast packets for sure on ESX. That would be very counter productive.
Still puzzled - but you could be right - it has something to do with ARP.
Regards
ilya
-----Original Message-----
From: Marcus Sorensen [mailto:shadowsor@gmail.com]
Sent: Monday, December 03, 2012 2:12 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: Router VM and Network Issue
I don't have a vsphere environment, but it sounds a bit like an ARP issue, where perhaps the bridge on the VMware host is filtering broadcast or something. Maybe a workaround could be added to the system VM to make sure things are more robust, but it sounds like something in your network is disabling the discovery of your router VM via broadcast, and when you ping from the system vm then the MAC table on the switch is finally updated and things start working.
On Dec 3, 2012 11:59 AM, "Musayev, Ilya" <im...@webmd.net> wrote:
> So I hit a glitch where a router VM boots up but does not really pass
> any traffic unless I ping the gateway of the CS host from within the router VM.
>
> Once the gateway ping goes through, CS is able to SSH into a router VM
> and everything is fine and dandy..
>
> But this behavior really puzzles me. Linux network stack is not fully
> activated or routing is not fully functional until the initial CS GW ping.
>
> Also I cant ping/ssh the router VM from CS unless a initiate a ping
> from within the router VM.
>
> I'm on CS 4.0 and vSphere5. This seem to affect the Advanced Network
> setup more than Basic because of routing complexity - as you add some
> routes into linux routing table.
>
>
> Has anyone seen this before?
>
Re: Router VM and Network Issue
Posted by Marcus Sorensen <sh...@gmail.com>.
I don't have a vsphere environment, but it sounds a bit like an ARP issue,
where perhaps the bridge on the VMware host is filtering broadcast or
something. Maybe a workaround could be added to the system VM to make sure
things are more robust, but it sounds like something in your network is
disabling the discovery of your router VM via broadcast, and when you ping
from the system vm then the MAC table on the switch is finally updated and
things start working.
On Dec 3, 2012 11:59 AM, "Musayev, Ilya" <im...@webmd.net> wrote:
> So I hit a glitch where a router VM boots up but does not really pass any
> traffic unless I ping the gateway of the CS host from within the router VM.
>
> Once the gateway ping goes through, CS is able to SSH into a router VM and
> everything is fine and dandy..
>
> But this behavior really puzzles me. Linux network stack is not fully
> activated or routing is not fully functional until the initial CS GW ping.
>
> Also I cant ping/ssh the router VM from CS unless a initiate a ping from
> within the router VM.
>
> I'm on CS 4.0 and vSphere5. This seem to affect the Advanced Network setup
> more than Basic because of routing complexity - as you add some routes into
> linux routing table.
>
>
> Has anyone seen this before?
>
Re: Router VM and Network Issue
Posted by "Musayev, Ilya" <im...@webmd.net>.
No Arista switch.. we do have junipers. My solution for now is to add a ping to a gateway of CS into cloud-early-config.
I will respond with network details tomorrow.
Thank you for your help
Ilya
Chiradeep Vittal <Ch...@citrix.com> wrote:
Is there an Arista switch in the path by any chance?
On 12/3/12 2:44 PM, "Anthony Xu" <Xu...@citrix.com> wrote:
>> What is the point of adding this extra route - if external routing
>> handles that by default?
>The extra route is added to make sure management server can talk to route
>VM.
>
>
>Can you share your setup info?
>Zone setup, network type, Private IP range, public IP range, VLAN info.
>
>
>This issue might happen when private ip CIDR overlaps with public ip
>CIDR, CS might not check this case.
>
>Anthony
>
>
>
>> -----Original Message-----
>> From: Musayev, Ilya [mailto:imusayev@webmd.net]
>> Sent: Monday, December 03, 2012 2:15 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: RE: Router VM and Network Issue
>>
>> Let me retract this comment for now and do more thorough testing.. It
>> appears it fixes the issue on 1 type of network and breaks on another...
>>
>> -----Original Message-----
>> From: Musayev, Ilya [mailto:imusayev@webmd.net]
>> Sent: Monday, December 03, 2012 4:43 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: RE: Router VM and Network Issue
>>
>> Anthony,
>>
>> I do have the code below, but my fix was to remove the extra route that
>> is added by command
>>
>> "ip route add $MGMTNET via $LOCAL_GW dev eth1" from cloud-early-config
>>
>> Once I commented that part out - everything is working fine..
>>
>> What is the point of adding this extra route - if external routing
>> handles that by default?
>>
>> In my case - the "route add" created ARP issues.
>>
>> Thank you for very helpful feedback
>> -ilya
>>
>>
>> -----Original Message-----
>> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
>> Sent: Monday, December 03, 2012 4:08 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: RE: Router VM and Network Issue
>>
>> I checked the code
>> in ./patches/systemvm/debian/config/etc/init.d/cloud-early-config
>>
>> # a hacking way to activate vSwitch under VMware
>> ping -n -c 3 $GW &
>> sleep 3
>> pkill ping
>> if [ -n "$MGMTNET" -a -n "$LOCAL_GW" ]
>> then
>> ping -n -c 3 $LOCAL_GW &
>> sleep 3
>> pkill ping
>> fi
>>
>> It pings both local and public gateway,
>>
>> Could you check the file in your setup to see if the fix is in?
>>
>> Below is the procedure to fix the issue in your setup,
>>
>> 1.Check in the fix
>> 2. get the latest build
>> 3. upgrade your current setup
>> 4. stop/start all system VM (SSVM , CPVM, router VM)
>>
>>
>> Anthony
>>
>>
>>
>>
>>
>> > -----Original Message-----
>> > From: Musayev, Ilya [mailto:imusayev@webmd.net]
>> > Sent: Monday, December 03, 2012 12:37 PM
>> > To: cloudstack-dev@incubator.apache.org
>> > Subject: RE: Router VM and Network Issue
>> >
>> > Anthony,
>> >
>> > It does ping the local gateways and I can see that happening when
>> > router VM boots up.
>> >
>> > But the fix is to ping either CS Core or CS gateway - that truly
>> > addresses the issue.
>> >
>> > Any thoughts of how I can create this behavior in reproducible
>> fashion
>> > for all new routers?
>> >
>> > Thanks
>> > ilya
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
>> > Sent: Monday, December 03, 2012 3:12 PM
>> > To: cloudstack-dev@incubator.apache.org
>> > Subject: RE: Router VM and Network Issue
>> >
>> > I remember we have a fix for this, when route VM boots up, it tries
>> to
>> > ping default gateway to propagate its MAC to switch.
>> >
>> > Might be this fix is not checked into CS 4.0
>> >
>> > Anthony
>> >
>> > > -----Original Message-----
>> > > From: Musayev, Ilya [mailto:imusayev@webmd.net]
>> > > Sent: Monday, December 03, 2012 10:59 AM
>> > > To: cloudstack-dev@incubator.apache.org
>> > > Subject: Router VM and Network Issue
>> > >
>> > > So I hit a glitch where a router VM boots up but does not really
>> > > pass any traffic unless I ping the gateway of the CS host from
>> > > within the router VM.
>> > >
>> > > Once the gateway ping goes through, CS is able to SSH into a router
>> > VM
>> > > and everything is fine and dandy..
>> > >
>> > > But this behavior really puzzles me. Linux network stack is not
>> > > fully activated or routing is not fully functional until the
>> initial
>> > > CS GW ping.
>> > >
>> > > Also I cant ping/ssh the router VM from CS unless a initiate a ping
>> > > from within the router VM.
>> > >
>> > > I'm on CS 4.0 and vSphere5. This seem to affect the Advanced
>> Network
>> > > setup more than Basic because of routing complexity - as you add
>> > > some routes into linux routing table.
>> > >
>> > >
>> > > Has anyone seen this before?
>> >
>>
>>
>>
>>
>
Re: Router VM and Network Issue
Posted by Chiradeep Vittal <Ch...@citrix.com>.
Yes, by default the ARP timers in their gateways are set very long. If a
VM is destroyed and a new VM picks up the old IP (but with a new mac) the
router continues to send traffic to the old mac. The fix is to set the arp
timeout very low.
On 12/10/12 10:46 AM, "Chip Childers" <ch...@sungard.com> wrote:
>On Mon, Dec 3, 2012 at 9:29 PM, Chiradeep Vittal
><Ch...@citrix.com> wrote:
>> Is there an Arista switch in the path by any chance?
>
>Out of curiosity, why do you ask Chiradeep? Have you run into
>oddities with them?
>
>> On 12/3/12 2:44 PM, "Anthony Xu" <Xu...@citrix.com> wrote:
>>
>>>> What is the point of adding this extra route - if external routing
>>>> handles that by default?
>>>The extra route is added to make sure management server can talk to
>>>route
>>>VM.
>>>
>>>
>>>Can you share your setup info?
>>>Zone setup, network type, Private IP range, public IP range, VLAN info.
>>>
>>>
>>>This issue might happen when private ip CIDR overlaps with public ip
>>>CIDR, CS might not check this case.
>>>
>>>Anthony
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: Musayev, Ilya [mailto:imusayev@webmd.net]
>>>> Sent: Monday, December 03, 2012 2:15 PM
>>>> To: cloudstack-dev@incubator.apache.org
>>>> Subject: RE: Router VM and Network Issue
>>>>
>>>> Let me retract this comment for now and do more thorough testing.. It
>>>> appears it fixes the issue on 1 type of network and breaks on
>>>>another...
>>>>
>>>> -----Original Message-----
>>>> From: Musayev, Ilya [mailto:imusayev@webmd.net]
>>>> Sent: Monday, December 03, 2012 4:43 PM
>>>> To: cloudstack-dev@incubator.apache.org
>>>> Subject: RE: Router VM and Network Issue
>>>>
>>>> Anthony,
>>>>
>>>> I do have the code below, but my fix was to remove the extra route
>>>>that
>>>> is added by command
>>>>
>>>> "ip route add $MGMTNET via $LOCAL_GW dev eth1" from cloud-early-config
>>>>
>>>> Once I commented that part out - everything is working fine..
>>>>
>>>> What is the point of adding this extra route - if external routing
>>>> handles that by default?
>>>>
>>>> In my case - the "route add" created ARP issues.
>>>>
>>>> Thank you for very helpful feedback
>>>> -ilya
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
>>>> Sent: Monday, December 03, 2012 4:08 PM
>>>> To: cloudstack-dev@incubator.apache.org
>>>> Subject: RE: Router VM and Network Issue
>>>>
>>>> I checked the code
>>>> in ./patches/systemvm/debian/config/etc/init.d/cloud-early-config
>>>>
>>>> # a hacking way to activate vSwitch under VMware
>>>> ping -n -c 3 $GW &
>>>> sleep 3
>>>> pkill ping
>>>> if [ -n "$MGMTNET" -a -n "$LOCAL_GW" ]
>>>> then
>>>> ping -n -c 3 $LOCAL_GW &
>>>> sleep 3
>>>> pkill ping
>>>> fi
>>>>
>>>> It pings both local and public gateway,
>>>>
>>>> Could you check the file in your setup to see if the fix is in?
>>>>
>>>> Below is the procedure to fix the issue in your setup,
>>>>
>>>> 1.Check in the fix
>>>> 2. get the latest build
>>>> 3. upgrade your current setup
>>>> 4. stop/start all system VM (SSVM , CPVM, router VM)
>>>>
>>>>
>>>> Anthony
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> > -----Original Message-----
>>>> > From: Musayev, Ilya [mailto:imusayev@webmd.net]
>>>> > Sent: Monday, December 03, 2012 12:37 PM
>>>> > To: cloudstack-dev@incubator.apache.org
>>>> > Subject: RE: Router VM and Network Issue
>>>> >
>>>> > Anthony,
>>>> >
>>>> > It does ping the local gateways and I can see that happening when
>>>> > router VM boots up.
>>>> >
>>>> > But the fix is to ping either CS Core or CS gateway - that truly
>>>> > addresses the issue.
>>>> >
>>>> > Any thoughts of how I can create this behavior in reproducible
>>>> fashion
>>>> > for all new routers?
>>>> >
>>>> > Thanks
>>>> > ilya
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
>>>> > Sent: Monday, December 03, 2012 3:12 PM
>>>> > To: cloudstack-dev@incubator.apache.org
>>>> > Subject: RE: Router VM and Network Issue
>>>> >
>>>> > I remember we have a fix for this, when route VM boots up, it tries
>>>> to
>>>> > ping default gateway to propagate its MAC to switch.
>>>> >
>>>> > Might be this fix is not checked into CS 4.0
>>>> >
>>>> > Anthony
>>>> >
>>>> > > -----Original Message-----
>>>> > > From: Musayev, Ilya [mailto:imusayev@webmd.net]
>>>> > > Sent: Monday, December 03, 2012 10:59 AM
>>>> > > To: cloudstack-dev@incubator.apache.org
>>>> > > Subject: Router VM and Network Issue
>>>> > >
>>>> > > So I hit a glitch where a router VM boots up but does not really
>>>> > > pass any traffic unless I ping the gateway of the CS host from
>>>> > > within the router VM.
>>>> > >
>>>> > > Once the gateway ping goes through, CS is able to SSH into a
>>>>router
>>>> > VM
>>>> > > and everything is fine and dandy..
>>>> > >
>>>> > > But this behavior really puzzles me. Linux network stack is not
>>>> > > fully activated or routing is not fully functional until the
>>>> initial
>>>> > > CS GW ping.
>>>> > >
>>>> > > Also I cant ping/ssh the router VM from CS unless a initiate a
>>>>ping
>>>> > > from within the router VM.
>>>> > >
>>>> > > I'm on CS 4.0 and vSphere5. This seem to affect the Advanced
>>>> Network
>>>> > > setup more than Basic because of routing complexity - as you add
>>>> > > some routes into linux routing table.
>>>> > >
>>>> > >
>>>> > > Has anyone seen this before?
>>>> >
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
Re: Router VM and Network Issue
Posted by Chip Childers <ch...@sungard.com>.
On Mon, Dec 3, 2012 at 9:29 PM, Chiradeep Vittal
<Ch...@citrix.com> wrote:
> Is there an Arista switch in the path by any chance?
Out of curiosity, why do you ask Chiradeep? Have you run into
oddities with them?
> On 12/3/12 2:44 PM, "Anthony Xu" <Xu...@citrix.com> wrote:
>
>>> What is the point of adding this extra route - if external routing
>>> handles that by default?
>>The extra route is added to make sure management server can talk to route
>>VM.
>>
>>
>>Can you share your setup info?
>>Zone setup, network type, Private IP range, public IP range, VLAN info.
>>
>>
>>This issue might happen when private ip CIDR overlaps with public ip
>>CIDR, CS might not check this case.
>>
>>Anthony
>>
>>
>>
>>> -----Original Message-----
>>> From: Musayev, Ilya [mailto:imusayev@webmd.net]
>>> Sent: Monday, December 03, 2012 2:15 PM
>>> To: cloudstack-dev@incubator.apache.org
>>> Subject: RE: Router VM and Network Issue
>>>
>>> Let me retract this comment for now and do more thorough testing.. It
>>> appears it fixes the issue on 1 type of network and breaks on another...
>>>
>>> -----Original Message-----
>>> From: Musayev, Ilya [mailto:imusayev@webmd.net]
>>> Sent: Monday, December 03, 2012 4:43 PM
>>> To: cloudstack-dev@incubator.apache.org
>>> Subject: RE: Router VM and Network Issue
>>>
>>> Anthony,
>>>
>>> I do have the code below, but my fix was to remove the extra route that
>>> is added by command
>>>
>>> "ip route add $MGMTNET via $LOCAL_GW dev eth1" from cloud-early-config
>>>
>>> Once I commented that part out - everything is working fine..
>>>
>>> What is the point of adding this extra route - if external routing
>>> handles that by default?
>>>
>>> In my case - the "route add" created ARP issues.
>>>
>>> Thank you for very helpful feedback
>>> -ilya
>>>
>>>
>>> -----Original Message-----
>>> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
>>> Sent: Monday, December 03, 2012 4:08 PM
>>> To: cloudstack-dev@incubator.apache.org
>>> Subject: RE: Router VM and Network Issue
>>>
>>> I checked the code
>>> in ./patches/systemvm/debian/config/etc/init.d/cloud-early-config
>>>
>>> # a hacking way to activate vSwitch under VMware
>>> ping -n -c 3 $GW &
>>> sleep 3
>>> pkill ping
>>> if [ -n "$MGMTNET" -a -n "$LOCAL_GW" ]
>>> then
>>> ping -n -c 3 $LOCAL_GW &
>>> sleep 3
>>> pkill ping
>>> fi
>>>
>>> It pings both local and public gateway,
>>>
>>> Could you check the file in your setup to see if the fix is in?
>>>
>>> Below is the procedure to fix the issue in your setup,
>>>
>>> 1.Check in the fix
>>> 2. get the latest build
>>> 3. upgrade your current setup
>>> 4. stop/start all system VM (SSVM , CPVM, router VM)
>>>
>>>
>>> Anthony
>>>
>>>
>>>
>>>
>>>
>>> > -----Original Message-----
>>> > From: Musayev, Ilya [mailto:imusayev@webmd.net]
>>> > Sent: Monday, December 03, 2012 12:37 PM
>>> > To: cloudstack-dev@incubator.apache.org
>>> > Subject: RE: Router VM and Network Issue
>>> >
>>> > Anthony,
>>> >
>>> > It does ping the local gateways and I can see that happening when
>>> > router VM boots up.
>>> >
>>> > But the fix is to ping either CS Core or CS gateway - that truly
>>> > addresses the issue.
>>> >
>>> > Any thoughts of how I can create this behavior in reproducible
>>> fashion
>>> > for all new routers?
>>> >
>>> > Thanks
>>> > ilya
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
>>> > Sent: Monday, December 03, 2012 3:12 PM
>>> > To: cloudstack-dev@incubator.apache.org
>>> > Subject: RE: Router VM and Network Issue
>>> >
>>> > I remember we have a fix for this, when route VM boots up, it tries
>>> to
>>> > ping default gateway to propagate its MAC to switch.
>>> >
>>> > Might be this fix is not checked into CS 4.0
>>> >
>>> > Anthony
>>> >
>>> > > -----Original Message-----
>>> > > From: Musayev, Ilya [mailto:imusayev@webmd.net]
>>> > > Sent: Monday, December 03, 2012 10:59 AM
>>> > > To: cloudstack-dev@incubator.apache.org
>>> > > Subject: Router VM and Network Issue
>>> > >
>>> > > So I hit a glitch where a router VM boots up but does not really
>>> > > pass any traffic unless I ping the gateway of the CS host from
>>> > > within the router VM.
>>> > >
>>> > > Once the gateway ping goes through, CS is able to SSH into a router
>>> > VM
>>> > > and everything is fine and dandy..
>>> > >
>>> > > But this behavior really puzzles me. Linux network stack is not
>>> > > fully activated or routing is not fully functional until the
>>> initial
>>> > > CS GW ping.
>>> > >
>>> > > Also I cant ping/ssh the router VM from CS unless a initiate a ping
>>> > > from within the router VM.
>>> > >
>>> > > I'm on CS 4.0 and vSphere5. This seem to affect the Advanced
>>> Network
>>> > > setup more than Basic because of routing complexity - as you add
>>> > > some routes into linux routing table.
>>> > >
>>> > >
>>> > > Has anyone seen this before?
>>> >
>>>
>>>
>>>
>>>
>>
>
>
Re: Router VM and Network Issue
Posted by Chiradeep Vittal <Ch...@citrix.com>.
Is there an Arista switch in the path by any chance?
On 12/3/12 2:44 PM, "Anthony Xu" <Xu...@citrix.com> wrote:
>> What is the point of adding this extra route - if external routing
>> handles that by default?
>The extra route is added to make sure management server can talk to route
>VM.
>
>
>Can you share your setup info?
>Zone setup, network type, Private IP range, public IP range, VLAN info.
>
>
>This issue might happen when private ip CIDR overlaps with public ip
>CIDR, CS might not check this case.
>
>Anthony
>
>
>
>> -----Original Message-----
>> From: Musayev, Ilya [mailto:imusayev@webmd.net]
>> Sent: Monday, December 03, 2012 2:15 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: RE: Router VM and Network Issue
>>
>> Let me retract this comment for now and do more thorough testing.. It
>> appears it fixes the issue on 1 type of network and breaks on another...
>>
>> -----Original Message-----
>> From: Musayev, Ilya [mailto:imusayev@webmd.net]
>> Sent: Monday, December 03, 2012 4:43 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: RE: Router VM and Network Issue
>>
>> Anthony,
>>
>> I do have the code below, but my fix was to remove the extra route that
>> is added by command
>>
>> "ip route add $MGMTNET via $LOCAL_GW dev eth1" from cloud-early-config
>>
>> Once I commented that part out - everything is working fine..
>>
>> What is the point of adding this extra route - if external routing
>> handles that by default?
>>
>> In my case - the "route add" created ARP issues.
>>
>> Thank you for very helpful feedback
>> -ilya
>>
>>
>> -----Original Message-----
>> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
>> Sent: Monday, December 03, 2012 4:08 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: RE: Router VM and Network Issue
>>
>> I checked the code
>> in ./patches/systemvm/debian/config/etc/init.d/cloud-early-config
>>
>> # a hacking way to activate vSwitch under VMware
>> ping -n -c 3 $GW &
>> sleep 3
>> pkill ping
>> if [ -n "$MGMTNET" -a -n "$LOCAL_GW" ]
>> then
>> ping -n -c 3 $LOCAL_GW &
>> sleep 3
>> pkill ping
>> fi
>>
>> It pings both local and public gateway,
>>
>> Could you check the file in your setup to see if the fix is in?
>>
>> Below is the procedure to fix the issue in your setup,
>>
>> 1.Check in the fix
>> 2. get the latest build
>> 3. upgrade your current setup
>> 4. stop/start all system VM (SSVM , CPVM, router VM)
>>
>>
>> Anthony
>>
>>
>>
>>
>>
>> > -----Original Message-----
>> > From: Musayev, Ilya [mailto:imusayev@webmd.net]
>> > Sent: Monday, December 03, 2012 12:37 PM
>> > To: cloudstack-dev@incubator.apache.org
>> > Subject: RE: Router VM and Network Issue
>> >
>> > Anthony,
>> >
>> > It does ping the local gateways and I can see that happening when
>> > router VM boots up.
>> >
>> > But the fix is to ping either CS Core or CS gateway - that truly
>> > addresses the issue.
>> >
>> > Any thoughts of how I can create this behavior in reproducible
>> fashion
>> > for all new routers?
>> >
>> > Thanks
>> > ilya
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
>> > Sent: Monday, December 03, 2012 3:12 PM
>> > To: cloudstack-dev@incubator.apache.org
>> > Subject: RE: Router VM and Network Issue
>> >
>> > I remember we have a fix for this, when route VM boots up, it tries
>> to
>> > ping default gateway to propagate its MAC to switch.
>> >
>> > Might be this fix is not checked into CS 4.0
>> >
>> > Anthony
>> >
>> > > -----Original Message-----
>> > > From: Musayev, Ilya [mailto:imusayev@webmd.net]
>> > > Sent: Monday, December 03, 2012 10:59 AM
>> > > To: cloudstack-dev@incubator.apache.org
>> > > Subject: Router VM and Network Issue
>> > >
>> > > So I hit a glitch where a router VM boots up but does not really
>> > > pass any traffic unless I ping the gateway of the CS host from
>> > > within the router VM.
>> > >
>> > > Once the gateway ping goes through, CS is able to SSH into a router
>> > VM
>> > > and everything is fine and dandy..
>> > >
>> > > But this behavior really puzzles me. Linux network stack is not
>> > > fully activated or routing is not fully functional until the
>> initial
>> > > CS GW ping.
>> > >
>> > > Also I cant ping/ssh the router VM from CS unless a initiate a ping
>> > > from within the router VM.
>> > >
>> > > I'm on CS 4.0 and vSphere5. This seem to affect the Advanced
>> Network
>> > > setup more than Basic because of routing complexity - as you add
>> > > some routes into linux routing table.
>> > >
>> > >
>> > > Has anyone seen this before?
>> >
>>
>>
>>
>>
>
RE: Router VM and Network Issue
Posted by Anthony Xu <Xu...@citrix.com>.
> What is the point of adding this extra route - if external routing
> handles that by default?
The extra route is added to make sure management server can talk to route VM.
Can you share your setup info?
Zone setup, network type, Private IP range, public IP range, VLAN info.
This issue might happen when private ip CIDR overlaps with public ip CIDR, CS might not check this case.
Anthony
> -----Original Message-----
> From: Musayev, Ilya [mailto:imusayev@webmd.net]
> Sent: Monday, December 03, 2012 2:15 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Router VM and Network Issue
>
> Let me retract this comment for now and do more thorough testing.. It
> appears it fixes the issue on 1 type of network and breaks on another...
>
> -----Original Message-----
> From: Musayev, Ilya [mailto:imusayev@webmd.net]
> Sent: Monday, December 03, 2012 4:43 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Router VM and Network Issue
>
> Anthony,
>
> I do have the code below, but my fix was to remove the extra route that
> is added by command
>
> "ip route add $MGMTNET via $LOCAL_GW dev eth1" from cloud-early-config
>
> Once I commented that part out - everything is working fine..
>
> What is the point of adding this extra route - if external routing
> handles that by default?
>
> In my case - the "route add" created ARP issues.
>
> Thank you for very helpful feedback
> -ilya
>
>
> -----Original Message-----
> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> Sent: Monday, December 03, 2012 4:08 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Router VM and Network Issue
>
> I checked the code
> in ./patches/systemvm/debian/config/etc/init.d/cloud-early-config
>
> # a hacking way to activate vSwitch under VMware
> ping -n -c 3 $GW &
> sleep 3
> pkill ping
> if [ -n "$MGMTNET" -a -n "$LOCAL_GW" ]
> then
> ping -n -c 3 $LOCAL_GW &
> sleep 3
> pkill ping
> fi
>
> It pings both local and public gateway,
>
> Could you check the file in your setup to see if the fix is in?
>
> Below is the procedure to fix the issue in your setup,
>
> 1.Check in the fix
> 2. get the latest build
> 3. upgrade your current setup
> 4. stop/start all system VM (SSVM , CPVM, router VM)
>
>
> Anthony
>
>
>
>
>
> > -----Original Message-----
> > From: Musayev, Ilya [mailto:imusayev@webmd.net]
> > Sent: Monday, December 03, 2012 12:37 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Router VM and Network Issue
> >
> > Anthony,
> >
> > It does ping the local gateways and I can see that happening when
> > router VM boots up.
> >
> > But the fix is to ping either CS Core or CS gateway - that truly
> > addresses the issue.
> >
> > Any thoughts of how I can create this behavior in reproducible
> fashion
> > for all new routers?
> >
> > Thanks
> > ilya
> >
> >
> >
> > -----Original Message-----
> > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > Sent: Monday, December 03, 2012 3:12 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Router VM and Network Issue
> >
> > I remember we have a fix for this, when route VM boots up, it tries
> to
> > ping default gateway to propagate its MAC to switch.
> >
> > Might be this fix is not checked into CS 4.0
> >
> > Anthony
> >
> > > -----Original Message-----
> > > From: Musayev, Ilya [mailto:imusayev@webmd.net]
> > > Sent: Monday, December 03, 2012 10:59 AM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: Router VM and Network Issue
> > >
> > > So I hit a glitch where a router VM boots up but does not really
> > > pass any traffic unless I ping the gateway of the CS host from
> > > within the router VM.
> > >
> > > Once the gateway ping goes through, CS is able to SSH into a router
> > VM
> > > and everything is fine and dandy..
> > >
> > > But this behavior really puzzles me. Linux network stack is not
> > > fully activated or routing is not fully functional until the
> initial
> > > CS GW ping.
> > >
> > > Also I cant ping/ssh the router VM from CS unless a initiate a ping
> > > from within the router VM.
> > >
> > > I'm on CS 4.0 and vSphere5. This seem to affect the Advanced
> Network
> > > setup more than Basic because of routing complexity - as you add
> > > some routes into linux routing table.
> > >
> > >
> > > Has anyone seen this before?
> >
>
>
>
>
RE: Router VM and Network Issue
Posted by "Musayev, Ilya" <im...@webmd.net>.
Let me retract this comment for now and do more thorough testing.. It appears it fixes the issue on 1 type of network and breaks on another...
-----Original Message-----
From: Musayev, Ilya [mailto:imusayev@webmd.net]
Sent: Monday, December 03, 2012 4:43 PM
To: cloudstack-dev@incubator.apache.org
Subject: RE: Router VM and Network Issue
Anthony,
I do have the code below, but my fix was to remove the extra route that is added by command
"ip route add $MGMTNET via $LOCAL_GW dev eth1" from cloud-early-config
Once I commented that part out - everything is working fine..
What is the point of adding this extra route - if external routing handles that by default?
In my case - the "route add" created ARP issues.
Thank you for very helpful feedback
-ilya
-----Original Message-----
From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
Sent: Monday, December 03, 2012 4:08 PM
To: cloudstack-dev@incubator.apache.org
Subject: RE: Router VM and Network Issue
I checked the code in ./patches/systemvm/debian/config/etc/init.d/cloud-early-config
# a hacking way to activate vSwitch under VMware
ping -n -c 3 $GW &
sleep 3
pkill ping
if [ -n "$MGMTNET" -a -n "$LOCAL_GW" ]
then
ping -n -c 3 $LOCAL_GW &
sleep 3
pkill ping
fi
It pings both local and public gateway,
Could you check the file in your setup to see if the fix is in?
Below is the procedure to fix the issue in your setup,
1.Check in the fix
2. get the latest build
3. upgrade your current setup
4. stop/start all system VM (SSVM , CPVM, router VM)
Anthony
> -----Original Message-----
> From: Musayev, Ilya [mailto:imusayev@webmd.net]
> Sent: Monday, December 03, 2012 12:37 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Router VM and Network Issue
>
> Anthony,
>
> It does ping the local gateways and I can see that happening when
> router VM boots up.
>
> But the fix is to ping either CS Core or CS gateway - that truly
> addresses the issue.
>
> Any thoughts of how I can create this behavior in reproducible fashion
> for all new routers?
>
> Thanks
> ilya
>
>
>
> -----Original Message-----
> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> Sent: Monday, December 03, 2012 3:12 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Router VM and Network Issue
>
> I remember we have a fix for this, when route VM boots up, it tries to
> ping default gateway to propagate its MAC to switch.
>
> Might be this fix is not checked into CS 4.0
>
> Anthony
>
> > -----Original Message-----
> > From: Musayev, Ilya [mailto:imusayev@webmd.net]
> > Sent: Monday, December 03, 2012 10:59 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Router VM and Network Issue
> >
> > So I hit a glitch where a router VM boots up but does not really
> > pass any traffic unless I ping the gateway of the CS host from
> > within the router VM.
> >
> > Once the gateway ping goes through, CS is able to SSH into a router
> VM
> > and everything is fine and dandy..
> >
> > But this behavior really puzzles me. Linux network stack is not
> > fully activated or routing is not fully functional until the initial
> > CS GW ping.
> >
> > Also I cant ping/ssh the router VM from CS unless a initiate a ping
> > from within the router VM.
> >
> > I'm on CS 4.0 and vSphere5. This seem to affect the Advanced Network
> > setup more than Basic because of routing complexity - as you add
> > some routes into linux routing table.
> >
> >
> > Has anyone seen this before?
>
RE: Router VM and Network Issue
Posted by "Musayev, Ilya" <im...@webmd.net>.
Anthony,
I do have the code below, but my fix was to remove the extra route that is added by command
"ip route add $MGMTNET via $LOCAL_GW dev eth1" from cloud-early-config
Once I commented that part out - everything is working fine..
What is the point of adding this extra route - if external routing handles that by default?
In my case - the "route add" created ARP issues.
Thank you for very helpful feedback
-ilya
-----Original Message-----
From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
Sent: Monday, December 03, 2012 4:08 PM
To: cloudstack-dev@incubator.apache.org
Subject: RE: Router VM and Network Issue
I checked the code in ./patches/systemvm/debian/config/etc/init.d/cloud-early-config
# a hacking way to activate vSwitch under VMware
ping -n -c 3 $GW &
sleep 3
pkill ping
if [ -n "$MGMTNET" -a -n "$LOCAL_GW" ]
then
ping -n -c 3 $LOCAL_GW &
sleep 3
pkill ping
fi
It pings both local and public gateway,
Could you check the file in your setup to see if the fix is in?
Below is the procedure to fix the issue in your setup,
1.Check in the fix
2. get the latest build
3. upgrade your current setup
4. stop/start all system VM (SSVM , CPVM, router VM)
Anthony
> -----Original Message-----
> From: Musayev, Ilya [mailto:imusayev@webmd.net]
> Sent: Monday, December 03, 2012 12:37 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Router VM and Network Issue
>
> Anthony,
>
> It does ping the local gateways and I can see that happening when
> router VM boots up.
>
> But the fix is to ping either CS Core or CS gateway - that truly
> addresses the issue.
>
> Any thoughts of how I can create this behavior in reproducible fashion
> for all new routers?
>
> Thanks
> ilya
>
>
>
> -----Original Message-----
> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> Sent: Monday, December 03, 2012 3:12 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Router VM and Network Issue
>
> I remember we have a fix for this, when route VM boots up, it tries to
> ping default gateway to propagate its MAC to switch.
>
> Might be this fix is not checked into CS 4.0
>
> Anthony
>
> > -----Original Message-----
> > From: Musayev, Ilya [mailto:imusayev@webmd.net]
> > Sent: Monday, December 03, 2012 10:59 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Router VM and Network Issue
> >
> > So I hit a glitch where a router VM boots up but does not really
> > pass any traffic unless I ping the gateway of the CS host from
> > within the router VM.
> >
> > Once the gateway ping goes through, CS is able to SSH into a router
> VM
> > and everything is fine and dandy..
> >
> > But this behavior really puzzles me. Linux network stack is not
> > fully activated or routing is not fully functional until the initial
> > CS GW ping.
> >
> > Also I cant ping/ssh the router VM from CS unless a initiate a ping
> > from within the router VM.
> >
> > I'm on CS 4.0 and vSphere5. This seem to affect the Advanced Network
> > setup more than Basic because of routing complexity - as you add
> > some routes into linux routing table.
> >
> >
> > Has anyone seen this before?
>
RE: Router VM and Network Issue
Posted by Anthony Xu <Xu...@citrix.com>.
I checked the code in ./patches/systemvm/debian/config/etc/init.d/cloud-early-config
# a hacking way to activate vSwitch under VMware
ping -n -c 3 $GW &
sleep 3
pkill ping
if [ -n "$MGMTNET" -a -n "$LOCAL_GW" ]
then
ping -n -c 3 $LOCAL_GW &
sleep 3
pkill ping
fi
It pings both local and public gateway,
Could you check the file in your setup to see if the fix is in?
Below is the procedure to fix the issue in your setup,
1.Check in the fix
2. get the latest build
3. upgrade your current setup
4. stop/start all system VM (SSVM , CPVM, router VM)
Anthony
> -----Original Message-----
> From: Musayev, Ilya [mailto:imusayev@webmd.net]
> Sent: Monday, December 03, 2012 12:37 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Router VM and Network Issue
>
> Anthony,
>
> It does ping the local gateways and I can see that happening when
> router VM boots up.
>
> But the fix is to ping either CS Core or CS gateway - that truly
> addresses the issue.
>
> Any thoughts of how I can create this behavior in reproducible fashion
> for all new routers?
>
> Thanks
> ilya
>
>
>
> -----Original Message-----
> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> Sent: Monday, December 03, 2012 3:12 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Router VM and Network Issue
>
> I remember we have a fix for this, when route VM boots up, it tries to
> ping default gateway to propagate its MAC to switch.
>
> Might be this fix is not checked into CS 4.0
>
> Anthony
>
> > -----Original Message-----
> > From: Musayev, Ilya [mailto:imusayev@webmd.net]
> > Sent: Monday, December 03, 2012 10:59 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Router VM and Network Issue
> >
> > So I hit a glitch where a router VM boots up but does not really pass
> > any traffic unless I ping the gateway of the CS host from within the
> > router VM.
> >
> > Once the gateway ping goes through, CS is able to SSH into a router
> VM
> > and everything is fine and dandy..
> >
> > But this behavior really puzzles me. Linux network stack is not fully
> > activated or routing is not fully functional until the initial CS GW
> > ping.
> >
> > Also I cant ping/ssh the router VM from CS unless a initiate a ping
> > from within the router VM.
> >
> > I'm on CS 4.0 and vSphere5. This seem to affect the Advanced Network
> > setup more than Basic because of routing complexity - as you add some
> > routes into linux routing table.
> >
> >
> > Has anyone seen this before?
>
RE: Router VM and Network Issue
Posted by "Musayev, Ilya" <im...@webmd.net>.
Anthony,
It does ping the local gateways and I can see that happening when router VM boots up.
But the fix is to ping either CS Core or CS gateway - that truly addresses the issue.
Any thoughts of how I can create this behavior in reproducible fashion for all new routers?
Thanks
ilya
-----Original Message-----
From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
Sent: Monday, December 03, 2012 3:12 PM
To: cloudstack-dev@incubator.apache.org
Subject: RE: Router VM and Network Issue
I remember we have a fix for this, when route VM boots up, it tries to ping default gateway to propagate its MAC to switch.
Might be this fix is not checked into CS 4.0
Anthony
> -----Original Message-----
> From: Musayev, Ilya [mailto:imusayev@webmd.net]
> Sent: Monday, December 03, 2012 10:59 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Router VM and Network Issue
>
> So I hit a glitch where a router VM boots up but does not really pass
> any traffic unless I ping the gateway of the CS host from within the
> router VM.
>
> Once the gateway ping goes through, CS is able to SSH into a router VM
> and everything is fine and dandy..
>
> But this behavior really puzzles me. Linux network stack is not fully
> activated or routing is not fully functional until the initial CS GW
> ping.
>
> Also I cant ping/ssh the router VM from CS unless a initiate a ping
> from within the router VM.
>
> I'm on CS 4.0 and vSphere5. This seem to affect the Advanced Network
> setup more than Basic because of routing complexity - as you add some
> routes into linux routing table.
>
>
> Has anyone seen this before?
RE: Router VM and Network Issue
Posted by Anthony Xu <Xu...@citrix.com>.
I remember we have a fix for this, when route VM boots up, it tries to ping default gateway to propagate its MAC to switch.
Might be this fix is not checked into CS 4.0
Anthony
> -----Original Message-----
> From: Musayev, Ilya [mailto:imusayev@webmd.net]
> Sent: Monday, December 03, 2012 10:59 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Router VM and Network Issue
>
> So I hit a glitch where a router VM boots up but does not really pass
> any traffic unless I ping the gateway of the CS host from within the
> router VM.
>
> Once the gateway ping goes through, CS is able to SSH into a router VM
> and everything is fine and dandy..
>
> But this behavior really puzzles me. Linux network stack is not fully
> activated or routing is not fully functional until the initial CS GW
> ping.
>
> Also I cant ping/ssh the router VM from CS unless a initiate a ping
> from within the router VM.
>
> I'm on CS 4.0 and vSphere5. This seem to affect the Advanced Network
> setup more than Basic because of routing complexity - as you add some
> routes into linux routing table.
>
>
> Has anyone seen this before?