You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Chandler Li <li...@gmail.com> on 2013/12/11 09:16:44 UTC
[Help] OVS Tunnel ARP reply storm problem
Hi,
I am trying to learn the procedure of multi-tenancy network created by GRE
tunnel in CloudStack and trying to establish one in Xenserver 6.2
environment step by step.
According to the sourcecode in ovstunnel, CloudStack need to prevent
broadcast storm by generating some flow rules on openvswitch. I create a
full mash topology using GRE tunnel, create some VMs, and set broadcast
storm prevent rules on each ovs. Unfortunately, amount of ARP reply storm
packet flow between all hosts after one VM pings the other VM in another
Host. (https://www.dropbox.com/s/vy1opm7plho9dzs/arp%20reply%20storm.PNG)
In this case, IP 10.0.0.1's VM is on IP 10.0.75.9's Host, IP 10.0.0.2's VM
is on IP 10.0.75.14's Host. The ARP request broadcast packets were dropped
successfully by the broadcast storm prevent rules excepted the real target
IP 10.0.75.14's Host, but ARP reply packets seems cannot be filtered by the
rules, so the ARP reply storm happened in this loop topology.
Could anyone tell me how CloudStack ovs tunnel avoid ARP reply storm or did
I miss something important? Thanks!
BRs,
Chandler Li
Re: [Help] OVS Tunnel ARP reply storm problem
Posted by Nguyen Anh Tu <tu...@apache.org>.
I filed the bug here: https://issues.apache.org/jira/browse/CLOUDSTACK-5516
Need anyone who master in networking can help. Temporary I still let
enabled STP. Or we can go back to disabled STP if needed.
Cheers,
--Tuna
On Wed, Dec 11, 2013 at 7:10 PM, Murali Reddy <Mu...@citrix.com>wrote:
> Tuna,
>
> I see you enabled STP on the OVS bridge. I believe STP was intentionally
> disabled for similar concerns of Chandler. Please see [1] for how the
> broadcast traffic was to be handled in non loop free full-mesh topology.
>
> On 11/12/13 3:07 PM, "Chandler Li" <li...@gmail.com> wrote:
>
> >Hi Tuna,
> >
> >Thanks for your reply. It's really helpful to me!
> >
> >Active the STP function on ovs bridge will prevent the storm effectively
> >but it will need large of CPU resource in a large amount of networks
> >scenario. Maybe there are other solutions to solve it easily? (Might be
> >adding more flow rules or else.)
>
> Chandler,
>
> Propagation of broadcast traffic across the tunnels should have been
> suppressed. Its possible there is a bug, could you file a bug? Generally
> effective handling of broadcast traffic is in TODO's [2]
>
> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/OVS+Tunnel+Manager+f
> or+CloudStack
> [2]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Enhancements+to+GRE-
> based+SDN+overlay
>
>
> >
> >Chandler
> >
> >
> >2013/12/11 Nguyen Anh Tu <tu...@apache.org>
> >
> >> Hi Chandler,
> >>
> >> This update will help you prevent the storm.
> >>
> >>
> >>
> >>
> http://blog.scottlowe.org/2013/11/22/an-update-on-using-gre-tunnels-with-
> >>open-vswitch/
> >>
> >> I'm taking care about GRE controller in CloudStack. Checking lastest
> >>master
> >> branch now.
> >>
> >> Cheers,
> >>
> >> --Tuna
> >>
> >>
> >> On Wed, Dec 11, 2013 at 3:16 PM, Chandler Li <lichandler116@gmail.com
> >> >wrote:
> >>
> >> > Hi,
> >> >
> >> > I am trying to learn the procedure of multi-tenancy network created by
> >> GRE
> >> > tunnel in CloudStack and trying to establish one in Xenserver 6.2
> >> > environment step by step.
> >> >
> >> > According to the sourcecode in ovstunnel, CloudStack need to prevent
> >> > broadcast storm by generating some flow rules on openvswitch. I
> >>create a
> >> > full mash topology using GRE tunnel, create some VMs, and set
> >>broadcast
> >> > storm prevent rules on each ovs. Unfortunately, amount of ARP reply
> >> storm
> >> > packet flow between all hosts after one VM pings the other VM in
> >>another
> >> > Host.
> >>(https://www.dropbox.com/s/vy1opm7plho9dzs/arp%20reply%20storm.PNG
> >> )
> >> >
> >> > In this case, IP 10.0.0.1's VM is on IP 10.0.75.9's Host, IP
> >>10.0.0.2's
> >> VM
> >> > is on IP 10.0.75.14's Host. The ARP request broadcast packets were
> >> dropped
> >> > successfully by the broadcast storm prevent rules excepted the real
> >> target
> >> > IP 10.0.75.14's Host, but ARP reply packets seems cannot be filtered
> >>by
> >> the
> >> > rules, so the ARP reply storm happened in this loop topology.
> >> >
> >> > Could anyone tell me how CloudStack ovs tunnel avoid ARP reply storm
> >>or
> >> did
> >> > I miss something important? Thanks!
> >> >
> >> > BRs,
> >> > Chandler Li
> >> >
> >>
> >
>
>
>
Re: [Help] OVS Tunnel ARP reply storm problem
Posted by Murali Reddy <Mu...@citrix.com>.
Tuna,
I see you enabled STP on the OVS bridge. I believe STP was intentionally
disabled for similar concerns of Chandler. Please see [1] for how the
broadcast traffic was to be handled in non loop free full-mesh topology.
On 11/12/13 3:07 PM, "Chandler Li" <li...@gmail.com> wrote:
>Hi Tuna,
>
>Thanks for your reply. It's really helpful to me!
>
>Active the STP function on ovs bridge will prevent the storm effectively
>but it will need large of CPU resource in a large amount of networks
>scenario. Maybe there are other solutions to solve it easily? (Might be
>adding more flow rules or else.)
Chandler,
Propagation of broadcast traffic across the tunnels should have been
suppressed. Its possible there is a bug, could you file a bug? Generally
effective handling of broadcast traffic is in TODO's [2]
[1]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/OVS+Tunnel+Manager+f
or+CloudStack
[2]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Enhancements+to+GRE-
based+SDN+overlay
>
>Chandler
>
>
>2013/12/11 Nguyen Anh Tu <tu...@apache.org>
>
>> Hi Chandler,
>>
>> This update will help you prevent the storm.
>>
>>
>>
>>http://blog.scottlowe.org/2013/11/22/an-update-on-using-gre-tunnels-with-
>>open-vswitch/
>>
>> I'm taking care about GRE controller in CloudStack. Checking lastest
>>master
>> branch now.
>>
>> Cheers,
>>
>> --Tuna
>>
>>
>> On Wed, Dec 11, 2013 at 3:16 PM, Chandler Li <lichandler116@gmail.com
>> >wrote:
>>
>> > Hi,
>> >
>> > I am trying to learn the procedure of multi-tenancy network created by
>> GRE
>> > tunnel in CloudStack and trying to establish one in Xenserver 6.2
>> > environment step by step.
>> >
>> > According to the sourcecode in ovstunnel, CloudStack need to prevent
>> > broadcast storm by generating some flow rules on openvswitch. I
>>create a
>> > full mash topology using GRE tunnel, create some VMs, and set
>>broadcast
>> > storm prevent rules on each ovs. Unfortunately, amount of ARP reply
>> storm
>> > packet flow between all hosts after one VM pings the other VM in
>>another
>> > Host.
>>(https://www.dropbox.com/s/vy1opm7plho9dzs/arp%20reply%20storm.PNG
>> )
>> >
>> > In this case, IP 10.0.0.1's VM is on IP 10.0.75.9's Host, IP
>>10.0.0.2's
>> VM
>> > is on IP 10.0.75.14's Host. The ARP request broadcast packets were
>> dropped
>> > successfully by the broadcast storm prevent rules excepted the real
>> target
>> > IP 10.0.75.14's Host, but ARP reply packets seems cannot be filtered
>>by
>> the
>> > rules, so the ARP reply storm happened in this loop topology.
>> >
>> > Could anyone tell me how CloudStack ovs tunnel avoid ARP reply storm
>>or
>> did
>> > I miss something important? Thanks!
>> >
>> > BRs,
>> > Chandler Li
>> >
>>
>
Re: [Help] OVS Tunnel ARP reply storm problem
Posted by Chandler Li <li...@gmail.com>.
Hi Tuna,
Thanks for your reply. It's really helpful to me!
Active the STP function on ovs bridge will prevent the storm effectively
but it will need large of CPU resource in a large amount of networks
scenario. Maybe there are other solutions to solve it easily? (Might be
adding more flow rules or else.)
Chandler
2013/12/11 Nguyen Anh Tu <tu...@apache.org>
> Hi Chandler,
>
> This update will help you prevent the storm.
>
>
> http://blog.scottlowe.org/2013/11/22/an-update-on-using-gre-tunnels-with-open-vswitch/
>
> I'm taking care about GRE controller in CloudStack. Checking lastest master
> branch now.
>
> Cheers,
>
> --Tuna
>
>
> On Wed, Dec 11, 2013 at 3:16 PM, Chandler Li <lichandler116@gmail.com
> >wrote:
>
> > Hi,
> >
> > I am trying to learn the procedure of multi-tenancy network created by
> GRE
> > tunnel in CloudStack and trying to establish one in Xenserver 6.2
> > environment step by step.
> >
> > According to the sourcecode in ovstunnel, CloudStack need to prevent
> > broadcast storm by generating some flow rules on openvswitch. I create a
> > full mash topology using GRE tunnel, create some VMs, and set broadcast
> > storm prevent rules on each ovs. Unfortunately, amount of ARP reply
> storm
> > packet flow between all hosts after one VM pings the other VM in another
> > Host. (https://www.dropbox.com/s/vy1opm7plho9dzs/arp%20reply%20storm.PNG
> )
> >
> > In this case, IP 10.0.0.1's VM is on IP 10.0.75.9's Host, IP 10.0.0.2's
> VM
> > is on IP 10.0.75.14's Host. The ARP request broadcast packets were
> dropped
> > successfully by the broadcast storm prevent rules excepted the real
> target
> > IP 10.0.75.14's Host, but ARP reply packets seems cannot be filtered by
> the
> > rules, so the ARP reply storm happened in this loop topology.
> >
> > Could anyone tell me how CloudStack ovs tunnel avoid ARP reply storm or
> did
> > I miss something important? Thanks!
> >
> > BRs,
> > Chandler Li
> >
>
Re: [Help] OVS Tunnel ARP reply storm problem
Posted by Nguyen Anh Tu <tu...@apache.org>.
Hi Chandler,
This update will help you prevent the storm.
http://blog.scottlowe.org/2013/11/22/an-update-on-using-gre-tunnels-with-open-vswitch/
I'm taking care about GRE controller in CloudStack. Checking lastest master
branch now.
Cheers,
--Tuna
On Wed, Dec 11, 2013 at 3:16 PM, Chandler Li <li...@gmail.com>wrote:
> Hi,
>
> I am trying to learn the procedure of multi-tenancy network created by GRE
> tunnel in CloudStack and trying to establish one in Xenserver 6.2
> environment step by step.
>
> According to the sourcecode in ovstunnel, CloudStack need to prevent
> broadcast storm by generating some flow rules on openvswitch. I create a
> full mash topology using GRE tunnel, create some VMs, and set broadcast
> storm prevent rules on each ovs. Unfortunately, amount of ARP reply storm
> packet flow between all hosts after one VM pings the other VM in another
> Host. (https://www.dropbox.com/s/vy1opm7plho9dzs/arp%20reply%20storm.PNG)
>
> In this case, IP 10.0.0.1's VM is on IP 10.0.75.9's Host, IP 10.0.0.2's VM
> is on IP 10.0.75.14's Host. The ARP request broadcast packets were dropped
> successfully by the broadcast storm prevent rules excepted the real target
> IP 10.0.75.14's Host, but ARP reply packets seems cannot be filtered by the
> rules, so the ARP reply storm happened in this loop topology.
>
> Could anyone tell me how CloudStack ovs tunnel avoid ARP reply storm or did
> I miss something important? Thanks!
>
> BRs,
> Chandler Li
>