You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Jeff Hair <je...@greenqloud.com> on 2016/03/14 17:35:42 UTC
IPs Getting Stuck on Multiple Routers
Hi,
If you disassociate a bunch of public IPs at once from an account, it tends
to be that the IPs are still stuck on the virtual router even though they
have been successfully released on the management server. Another account
later acquires that IP, and now two routers have the same public IPs
attached to them. This leads to intermittent connectivity loss for the
accounts.
The only way currently to fix this is to restart network (not acceptable in
production environment) or to make some kind of cron job which manually
fixes broken IPs (barely any better).
This happens at least on KVM. I assume it happens on other hypervisors as
well.
Is this a known issue?
Thanks,
Jeff
Re: IPs Getting Stuck on Multiple Routers
Posted by Jeff Hair <je...@greenqloud.com>.
I created a new issue: https://issues.apache.org/jira/browse/CLOUDSTACK-9317
Also updated the pull request name with this ticket.
On Mon, Mar 21, 2016 at 5:06 PM, Simon Weller <sw...@ena.com> wrote:
> Jeff,
>
> Can you reference a jira issue in the subject of this pull request? All
> the comments on the pull will then be referenced against the issue. This
> makes reviewing easier.
>
> Thanks,
>
> - Si
>
>
> ________________________________________
> From: Jeff Hair <je...@greenqloud.com>
> Sent: Monday, March 21, 2016 12:00 PM
> To: users@cloudstack.apache.org
> Subject: Re: IPs Getting Stuck on Multiple Routers
>
> I have submitted a pull request to fix this issue on 4.7:
> https://github.com/apache/cloudstack/pull/1450
>
> Input/comments would be great.
>
> On Tue, Mar 15, 2016 at 1:18 PM, Jeff Hair <je...@greenqloud.com> wrote:
>
> > Don't think there's any real known workaround except a cron job to sync
> > the IPs on the router with what's in the database. This is how we solved
> it
> > in 4.2. Every 3 minutes get a list of IPs from the routers, compare to
> DB,
> > then send an IpAssoc command to fix them if necessary.
> >
> > But I'm looking for a better way to fix it this time. Maybe refactoring
> > the IP assignment in enableStaticNat to only associate/disassociate one
> IP
> > at a time would work.
> >
> > On Tue, Mar 15, 2016 at 9:26 AM, Sven Knohsalla <
> > sven.knohsalla@empolis.com> wrote:
> >
> >> Hi,
> >>
> >> we are running into the same problem.
> >> Is there are workaround for this issue ?
> >>
> >> Thanks in advance.
> >>
> >> Regards,
> >> Sven.
> >>
> >> -----Ursprüngliche Nachricht-----
> >> Von: Jeff Hair [mailto:jeff@greenqloud.com]
> >> Gesendet: Dienstag, 15. März 2016 09:49
> >> An: users@cloudstack.apache.org
> >> Betreff: Re: IPs Getting Stuck on Multiple Routers
> >>
> >> Thanks for the reply. Is there a JIRA ticket for this?
> >>
> >> On Tue, Mar 15, 2016 at 6:35 AM, Sanjeev Neelarapu <
> >> sanjeev.neelarapu@accelerite.com> wrote:
> >>
> >> > Hi Jeff,
> >> >
> >> > This is a known issue and is hypervisor agnostic.
> >> >
> >> > -Sanjeev
> >> >
> >> > -----Original Message-----
> >> > From: Jeff Hair [mailto:jeff@greenqloud.com]
> >> > Sent: Monday, March 14, 2016 10:06 PM
> >> > To: users@cloudstack.apache.org
> >> > Subject: IPs Getting Stuck on Multiple Routers
> >> >
> >> > Hi,
> >> >
> >> > If you disassociate a bunch of public IPs at once from an account, it
> >> > tends to be that the IPs are still stuck on the virtual router even
> >> > though they have been successfully released on the management server.
> >> > Another account later acquires that IP, and now two routers have the
> >> > same public IPs attached to them. This leads to intermittent
> >> > connectivity loss for the accounts.
> >> >
> >> > The only way currently to fix this is to restart network (not
> >> > acceptable in production environment) or to make some kind of cron job
> >> > which manually fixes broken IPs (barely any better).
> >> >
> >> > This happens at least on KVM. I assume it happens on other hypervisors
> >> > as well.
> >> >
> >> > Is this a known issue?
> >> >
> >> > Thanks,
> >> >
> >> > Jeff
> >> >
> >> >
> >> >
> >> > DISCLAIMER
> >> > ==========
> >> > This e-mail may contain privileged and confidential information which
> >> > is the property of Accelerite, a Persistent Systems business. It is
> >> > intended only for the use of the individual or entity to which it is
> >> > addressed. If you are not the intended recipient, you are not
> >> > authorized to read, retain, copy, print, distribute or use this
> >> > message. If you have received this communication in error, please
> >> > notify the sender and delete all copies of this message. Accelerite, a
> >> > Persistent Systems business does not accept any liability for virus
> >> infected mails.
> >> >
> >>
> >>
> >>
> >> --
> >> *Jeff Hair*
> >> Technical Lead and Software Developer
> >>
> >> Tel: (+354) 415 0200
> >> jeff@greenqloud.com
> >> www.greenqloud.com
> >>
> >
> >
> >
> > --
> > *Jeff Hair*
> > Technical Lead and Software Developer
> >
> > Tel: (+354) 415 0200
> > jeff@greenqloud.com
> > www.greenqloud.com
> >
> >
>
Re: IPs Getting Stuck on Multiple Routers
Posted by Simon Weller <sw...@ena.com>.
Jeff,
Can you reference a jira issue in the subject of this pull request? All the comments on the pull will then be referenced against the issue. This makes reviewing easier.
Thanks,
- Si
________________________________________
From: Jeff Hair <je...@greenqloud.com>
Sent: Monday, March 21, 2016 12:00 PM
To: users@cloudstack.apache.org
Subject: Re: IPs Getting Stuck on Multiple Routers
I have submitted a pull request to fix this issue on 4.7:
https://github.com/apache/cloudstack/pull/1450
Input/comments would be great.
On Tue, Mar 15, 2016 at 1:18 PM, Jeff Hair <je...@greenqloud.com> wrote:
> Don't think there's any real known workaround except a cron job to sync
> the IPs on the router with what's in the database. This is how we solved it
> in 4.2. Every 3 minutes get a list of IPs from the routers, compare to DB,
> then send an IpAssoc command to fix them if necessary.
>
> But I'm looking for a better way to fix it this time. Maybe refactoring
> the IP assignment in enableStaticNat to only associate/disassociate one IP
> at a time would work.
>
> On Tue, Mar 15, 2016 at 9:26 AM, Sven Knohsalla <
> sven.knohsalla@empolis.com> wrote:
>
>> Hi,
>>
>> we are running into the same problem.
>> Is there are workaround for this issue ?
>>
>> Thanks in advance.
>>
>> Regards,
>> Sven.
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Jeff Hair [mailto:jeff@greenqloud.com]
>> Gesendet: Dienstag, 15. März 2016 09:49
>> An: users@cloudstack.apache.org
>> Betreff: Re: IPs Getting Stuck on Multiple Routers
>>
>> Thanks for the reply. Is there a JIRA ticket for this?
>>
>> On Tue, Mar 15, 2016 at 6:35 AM, Sanjeev Neelarapu <
>> sanjeev.neelarapu@accelerite.com> wrote:
>>
>> > Hi Jeff,
>> >
>> > This is a known issue and is hypervisor agnostic.
>> >
>> > -Sanjeev
>> >
>> > -----Original Message-----
>> > From: Jeff Hair [mailto:jeff@greenqloud.com]
>> > Sent: Monday, March 14, 2016 10:06 PM
>> > To: users@cloudstack.apache.org
>> > Subject: IPs Getting Stuck on Multiple Routers
>> >
>> > Hi,
>> >
>> > If you disassociate a bunch of public IPs at once from an account, it
>> > tends to be that the IPs are still stuck on the virtual router even
>> > though they have been successfully released on the management server.
>> > Another account later acquires that IP, and now two routers have the
>> > same public IPs attached to them. This leads to intermittent
>> > connectivity loss for the accounts.
>> >
>> > The only way currently to fix this is to restart network (not
>> > acceptable in production environment) or to make some kind of cron job
>> > which manually fixes broken IPs (barely any better).
>> >
>> > This happens at least on KVM. I assume it happens on other hypervisors
>> > as well.
>> >
>> > Is this a known issue?
>> >
>> > Thanks,
>> >
>> > Jeff
>> >
>> >
>> >
>> > DISCLAIMER
>> > ==========
>> > This e-mail may contain privileged and confidential information which
>> > is the property of Accelerite, a Persistent Systems business. It is
>> > intended only for the use of the individual or entity to which it is
>> > addressed. If you are not the intended recipient, you are not
>> > authorized to read, retain, copy, print, distribute or use this
>> > message. If you have received this communication in error, please
>> > notify the sender and delete all copies of this message. Accelerite, a
>> > Persistent Systems business does not accept any liability for virus
>> infected mails.
>> >
>>
>>
>>
>> --
>> *Jeff Hair*
>> Technical Lead and Software Developer
>>
>> Tel: (+354) 415 0200
>> jeff@greenqloud.com
>> www.greenqloud.com
>>
>
>
>
> --
> *Jeff Hair*
> Technical Lead and Software Developer
>
> Tel: (+354) 415 0200
> jeff@greenqloud.com
> www.greenqloud.com
>
>
Re: IPs Getting Stuck on Multiple Routers
Posted by Jeff Hair <je...@greenqloud.com>.
I have submitted a pull request to fix this issue on 4.7:
https://github.com/apache/cloudstack/pull/1450
Input/comments would be great.
On Tue, Mar 15, 2016 at 1:18 PM, Jeff Hair <je...@greenqloud.com> wrote:
> Don't think there's any real known workaround except a cron job to sync
> the IPs on the router with what's in the database. This is how we solved it
> in 4.2. Every 3 minutes get a list of IPs from the routers, compare to DB,
> then send an IpAssoc command to fix them if necessary.
>
> But I'm looking for a better way to fix it this time. Maybe refactoring
> the IP assignment in enableStaticNat to only associate/disassociate one IP
> at a time would work.
>
> On Tue, Mar 15, 2016 at 9:26 AM, Sven Knohsalla <
> sven.knohsalla@empolis.com> wrote:
>
>> Hi,
>>
>> we are running into the same problem.
>> Is there are workaround for this issue ?
>>
>> Thanks in advance.
>>
>> Regards,
>> Sven.
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Jeff Hair [mailto:jeff@greenqloud.com]
>> Gesendet: Dienstag, 15. März 2016 09:49
>> An: users@cloudstack.apache.org
>> Betreff: Re: IPs Getting Stuck on Multiple Routers
>>
>> Thanks for the reply. Is there a JIRA ticket for this?
>>
>> On Tue, Mar 15, 2016 at 6:35 AM, Sanjeev Neelarapu <
>> sanjeev.neelarapu@accelerite.com> wrote:
>>
>> > Hi Jeff,
>> >
>> > This is a known issue and is hypervisor agnostic.
>> >
>> > -Sanjeev
>> >
>> > -----Original Message-----
>> > From: Jeff Hair [mailto:jeff@greenqloud.com]
>> > Sent: Monday, March 14, 2016 10:06 PM
>> > To: users@cloudstack.apache.org
>> > Subject: IPs Getting Stuck on Multiple Routers
>> >
>> > Hi,
>> >
>> > If you disassociate a bunch of public IPs at once from an account, it
>> > tends to be that the IPs are still stuck on the virtual router even
>> > though they have been successfully released on the management server.
>> > Another account later acquires that IP, and now two routers have the
>> > same public IPs attached to them. This leads to intermittent
>> > connectivity loss for the accounts.
>> >
>> > The only way currently to fix this is to restart network (not
>> > acceptable in production environment) or to make some kind of cron job
>> > which manually fixes broken IPs (barely any better).
>> >
>> > This happens at least on KVM. I assume it happens on other hypervisors
>> > as well.
>> >
>> > Is this a known issue?
>> >
>> > Thanks,
>> >
>> > Jeff
>> >
>> >
>> >
>> > DISCLAIMER
>> > ==========
>> > This e-mail may contain privileged and confidential information which
>> > is the property of Accelerite, a Persistent Systems business. It is
>> > intended only for the use of the individual or entity to which it is
>> > addressed. If you are not the intended recipient, you are not
>> > authorized to read, retain, copy, print, distribute or use this
>> > message. If you have received this communication in error, please
>> > notify the sender and delete all copies of this message. Accelerite, a
>> > Persistent Systems business does not accept any liability for virus
>> infected mails.
>> >
>>
>>
>>
>> --
>> *Jeff Hair*
>> Technical Lead and Software Developer
>>
>> Tel: (+354) 415 0200
>> jeff@greenqloud.com
>> www.greenqloud.com
>>
>
>
>
> --
> *Jeff Hair*
> Technical Lead and Software Developer
>
> Tel: (+354) 415 0200
> jeff@greenqloud.com
> www.greenqloud.com
>
>
Re: IPs Getting Stuck on Multiple Routers
Posted by Jeff Hair <je...@greenqloud.com>.
Don't think there's any real known workaround except a cron job to sync the
IPs on the router with what's in the database. This is how we solved it in
4.2. Every 3 minutes get a list of IPs from the routers, compare to DB,
then send an IpAssoc command to fix them if necessary.
But I'm looking for a better way to fix it this time. Maybe refactoring the
IP assignment in enableStaticNat to only associate/disassociate one IP at a
time would work.
On Tue, Mar 15, 2016 at 9:26 AM, Sven Knohsalla <sv...@empolis.com>
wrote:
> Hi,
>
> we are running into the same problem.
> Is there are workaround for this issue ?
>
> Thanks in advance.
>
> Regards,
> Sven.
>
> -----Ursprüngliche Nachricht-----
> Von: Jeff Hair [mailto:jeff@greenqloud.com]
> Gesendet: Dienstag, 15. März 2016 09:49
> An: users@cloudstack.apache.org
> Betreff: Re: IPs Getting Stuck on Multiple Routers
>
> Thanks for the reply. Is there a JIRA ticket for this?
>
> On Tue, Mar 15, 2016 at 6:35 AM, Sanjeev Neelarapu <
> sanjeev.neelarapu@accelerite.com> wrote:
>
> > Hi Jeff,
> >
> > This is a known issue and is hypervisor agnostic.
> >
> > -Sanjeev
> >
> > -----Original Message-----
> > From: Jeff Hair [mailto:jeff@greenqloud.com]
> > Sent: Monday, March 14, 2016 10:06 PM
> > To: users@cloudstack.apache.org
> > Subject: IPs Getting Stuck on Multiple Routers
> >
> > Hi,
> >
> > If you disassociate a bunch of public IPs at once from an account, it
> > tends to be that the IPs are still stuck on the virtual router even
> > though they have been successfully released on the management server.
> > Another account later acquires that IP, and now two routers have the
> > same public IPs attached to them. This leads to intermittent
> > connectivity loss for the accounts.
> >
> > The only way currently to fix this is to restart network (not
> > acceptable in production environment) or to make some kind of cron job
> > which manually fixes broken IPs (barely any better).
> >
> > This happens at least on KVM. I assume it happens on other hypervisors
> > as well.
> >
> > Is this a known issue?
> >
> > Thanks,
> >
> > Jeff
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which
> > is the property of Accelerite, a Persistent Systems business. It is
> > intended only for the use of the individual or entity to which it is
> > addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message. Accelerite, a
> > Persistent Systems business does not accept any liability for virus
> infected mails.
> >
>
>
>
> --
> *Jeff Hair*
> Technical Lead and Software Developer
>
> Tel: (+354) 415 0200
> jeff@greenqloud.com
> www.greenqloud.com
>
--
*Jeff Hair*
Technical Lead and Software Developer
Tel: (+354) 415 0200
jeff@greenqloud.com
www.greenqloud.com
AW: IPs Getting Stuck on Multiple Routers
Posted by Sven Knohsalla <sv...@empolis.com>.
Hi,
we are running into the same problem.
Is there are workaround for this issue ?
Thanks in advance.
Regards,
Sven.
-----Ursprüngliche Nachricht-----
Von: Jeff Hair [mailto:jeff@greenqloud.com]
Gesendet: Dienstag, 15. März 2016 09:49
An: users@cloudstack.apache.org
Betreff: Re: IPs Getting Stuck on Multiple Routers
Thanks for the reply. Is there a JIRA ticket for this?
On Tue, Mar 15, 2016 at 6:35 AM, Sanjeev Neelarapu < sanjeev.neelarapu@accelerite.com> wrote:
> Hi Jeff,
>
> This is a known issue and is hypervisor agnostic.
>
> -Sanjeev
>
> -----Original Message-----
> From: Jeff Hair [mailto:jeff@greenqloud.com]
> Sent: Monday, March 14, 2016 10:06 PM
> To: users@cloudstack.apache.org
> Subject: IPs Getting Stuck on Multiple Routers
>
> Hi,
>
> If you disassociate a bunch of public IPs at once from an account, it
> tends to be that the IPs are still stuck on the virtual router even
> though they have been successfully released on the management server.
> Another account later acquires that IP, and now two routers have the
> same public IPs attached to them. This leads to intermittent
> connectivity loss for the accounts.
>
> The only way currently to fix this is to restart network (not
> acceptable in production environment) or to make some kind of cron job
> which manually fixes broken IPs (barely any better).
>
> This happens at least on KVM. I assume it happens on other hypervisors
> as well.
>
> Is this a known issue?
>
> Thanks,
>
> Jeff
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which
> is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not
> authorized to read, retain, copy, print, distribute or use this
> message. If you have received this communication in error, please
> notify the sender and delete all copies of this message. Accelerite, a
> Persistent Systems business does not accept any liability for virus infected mails.
>
--
*Jeff Hair*
Technical Lead and Software Developer
Tel: (+354) 415 0200
jeff@greenqloud.com
www.greenqloud.com
Re: IPs Getting Stuck on Multiple Routers
Posted by Jeff Hair <je...@greenqloud.com>.
Thanks for the reply. Is there a JIRA ticket for this?
On Tue, Mar 15, 2016 at 6:35 AM, Sanjeev Neelarapu <
sanjeev.neelarapu@accelerite.com> wrote:
> Hi Jeff,
>
> This is a known issue and is hypervisor agnostic.
>
> -Sanjeev
>
> -----Original Message-----
> From: Jeff Hair [mailto:jeff@greenqloud.com]
> Sent: Monday, March 14, 2016 10:06 PM
> To: users@cloudstack.apache.org
> Subject: IPs Getting Stuck on Multiple Routers
>
> Hi,
>
> If you disassociate a bunch of public IPs at once from an account, it
> tends to be that the IPs are still stuck on the virtual router even though
> they have been successfully released on the management server. Another
> account later acquires that IP, and now two routers have the same public
> IPs attached to them. This leads to intermittent connectivity loss for the
> accounts.
>
> The only way currently to fix this is to restart network (not acceptable
> in production environment) or to make some kind of cron job which manually
> fixes broken IPs (barely any better).
>
> This happens at least on KVM. I assume it happens on other hypervisors as
> well.
>
> Is this a known issue?
>
> Thanks,
>
> Jeff
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>
--
*Jeff Hair*
Technical Lead and Software Developer
Tel: (+354) 415 0200
jeff@greenqloud.com
www.greenqloud.com
RE: IPs Getting Stuck on Multiple Routers
Posted by Sanjeev Neelarapu <sa...@accelerite.com>.
Hi Jeff,
This is a known issue and is hypervisor agnostic.
-Sanjeev
-----Original Message-----
From: Jeff Hair [mailto:jeff@greenqloud.com]
Sent: Monday, March 14, 2016 10:06 PM
To: users@cloudstack.apache.org
Subject: IPs Getting Stuck on Multiple Routers
Hi,
If you disassociate a bunch of public IPs at once from an account, it tends to be that the IPs are still stuck on the virtual router even though they have been successfully released on the management server. Another account later acquires that IP, and now two routers have the same public IPs attached to them. This leads to intermittent connectivity loss for the accounts.
The only way currently to fix this is to restart network (not acceptable in production environment) or to make some kind of cron job which manually fixes broken IPs (barely any better).
This happens at least on KVM. I assume it happens on other hypervisors as well.
Is this a known issue?
Thanks,
Jeff
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.