You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Daniel Hertanu <da...@unixmob.com> on 2013/09/23 16:08:20 UTC
Management IP address pool exhausted
Hi
I'm running CS 4.1.1 on CentOS 6.4 64 bit with one host Citrix Xenserver
6.1.
Few days ago I ran into a problem while I was configuring CS. The zone was
active, Xen host added, SSVM and CPVM up & running when someone from our
team rebooted the Xen host for whatever reason. Even though CloudStack was
complaining about the missing host in the logs, it deleted the old SSVM and
tried to deploy another one, it failed, then another one, and so on, until
the management IP addresses pool was exhausted, moment when CloudStack
reported the new error - that there are not enough management IPs. The host
came back online later but the situation didn't changed. I lowered the
expunge.delay and expunge.interval to 5 minutes, still no change.
I wiped out CS installation and I start all over again. I was curious about
the previous issue, so I manually rebooted the Xen host (without putting it
in Maintenance mode in CS) and the issue showed up again.
While I admit that having one host is not ideal, I have two questions:
- why would CS try to deploy any VM when it actually knows there are no
valid hosts
- how do I make those management IPs available again after CS tried to
deploy 10-20 SSVMs and it exhausted the management IPs pool?
Thanks,
Daniel
Re: Management IP address pool exhausted
Posted by Daniel Hertanu <da...@unixmob.com>.
I posted here: http://pastebin.com/uJ6jR73D the logs, maybe someone else
have seen this behavior before. Just to remind the issue, CS 4.1, XenServer
6.1, when the only existing host is rebooted without being in maintenance
mode (and stays down for a while), all the management IPs get used (in this
case there were 171 management IPs, all show as in use now).
Thanks,
Daniel
On Wed, Sep 25, 2013 at 8:56 PM, Musayev, Ilya <im...@webmd.net> wrote:
> Dan,
>
> Ping me on IRC, nick serverchief
>
> Regards
> ilya
>
>
> > -----Original Message-----
> > From: Daniel Hertanu [mailto:dart@unixmob.com]
> > Sent: Wednesday, September 25, 2013 6:44 AM
> > To: users@cloudstack.apache.org
> > Subject: Re: Management IP address pool exhausted
> >
> > Hi
> >
> > So I re-installed CS, reboot Xen host without entering it first in
> Maintenace
> > Mode, then got the management IP pool fully used. I followed again your
> > indications but to no avail.
> > In the meantime SSVM has reached number 112 (s-112-VM).
> >
> > I would really appreciate any other idea.
> >
> > Thanks,
> >
> > Daniel
> >
> >
> > On Wed, Sep 25, 2013 at 9:36 AM, Daniel Hertanu <da...@unixmob.com>
> > wrote:
> >
> > > Hello Ilya
> > >
> > > I submit an issue describing the bug:
> > > https://issues.apache.org/jira/browse/CLOUDSTACK-4735 (I picked CS
> > > version 4.1.0 as 4.1.1 is reported as non existent, but I've seen this
> > > behavior before since version 3).
> > >
> > > I followed your indications but it didn't work. After changing those
> > > records in the database, CS would still give the same error. I revert
> > > back to the initial db content, but this time nothing came back
> > > (initially the SSVM was still running, it was CPVM that didn't start).
> > > After this step I tried some tricks on my own and I ended up with a
> botched
> > environment.
> > > So I'll do a fresh install, reboot the host and try your suggestions
> > > again.
> > >
> > > Thanks,
> > >
> > > Daniel
> > >
> > >
> > > On Tue, Sep 24, 2013 at 10:46 PM, Musayev, Ilya
> > <im...@webmd.net>wrote:
> > >
> > >> I'm not certain if this is hypervisor specific - I've seen the same
> > >> issue for vSphere - but I though it was me, I think we may need to
> > >> revise the system vm ip space decommission process.
> > >>
> > >> Shanker and/or Daniel,
> > >>
> > >> Do you mind creating a ticket in issues.apache.org and write out ways
> > >> to reproduce this issue.
> > >>
> > >> ---
> > >>
> > >> *NOTE* this is purely from memory and some notes I had - proceed at
> > >> your own risk....
> > >>
> > >>
> > >>
> > >> In the meantime, lets clean up what you have.
> > >>
> > >> The goal is to release the IPs that are still marked as reserved and
> > >> allocated.
> > >>
> > >> I'd recommend installing mysql workbench
> > >>
> > >> There are few tables we need to review:
> > >> 1) Stop cloudstack management server
> > >> 2) Backup mysql db first.
> > >>
> > >> Note the id of management network referenced in cloud.networks
> > >> Select id from cloud.networks where traffic_type="Management"
> > >> In my case, Management network has ID of 205
> > >> SELECT * FROM cloud.nics where network_id=205; Alter to
> > >> Deallocating the ones that are marked as Reserved.
> > >>
> > >> Do the same for cloud.user_ip_address SELECT * FROM
> > >> cloud.user_ip_address where source_network_id=205; Alter from
> > >> Allocated to Free
> > >>
> > >> Stop any systemvms if running, and alter the state from Running to
> > >> Stopped or Expunging in cloud.vm_instance table. This will recreate
> > >> the systemvm from scratch.
> > >>
> > >> I think this should have you covered, if not, revert back the mysql
> db.
> > >> Do mysql dump and grep for IPs that should have been released.
> > >>
> > >> Let me know,
> > >>
> > >> Regards
> > >> ilya
> > >>
> > >>
> > >>
> > >> > -----Original Message-----
> > >> > From: Shanker Balan [mailto:shanker.balan@shapeblue.com]
> > >> > Sent: Tuesday, September 24, 2013 9:44 AM
> > >> > To: <us...@cloudstack.apache.org>
> > >> > Subject: Re: Management IP address pool exhausted
> > >> >
> > >> >
> > >> > On 24-Sep-2013, at 6:37 PM, Daniel Hertanu <da...@unixmob.com>
> > wrote:
> > >> >
> > >> > > Hello Ilya
> > >> > >
> > >> > > I repeated the issue by rebooting the Xen host without having it
> > >> > > in maintenance mode in CS. At this moment I have the SSVM up and
> > >> > > running and a virtual router (not related to the management IPs, I
> > know).
> > >> > > Console Proxy VM can't be started because:
> > >> > >
> > >> > > 2013-09-24 14:56:24,410 INFO
> > >> > > [cloud.vm.VirtualMachineManagerImpl]
> > >> > > (Job-Executor-22:job-72) Insufficient capacity
> > >> > > com.cloud.exception.InsufficientAddressCapacityException: Unable
> > >> > > to get a management ip addressScope=interface com.cloud.dc.Pod;
> > >> > > id=1
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > Hi Ilya,
> > >> >
> > >> > I have run into the same issue too. It is easily reproducible by
> > >> > having
> > >> only one
> > >> > host in the Zone and rebooting it without enabling maintenance mode
> > >> > on
> > >> it.
> > >> >
> > >> > This issue exists for both XenServer and KVM host when I tried.
> > >> >
> > >> >
> > >> > --
> > >> > @shankerbalan
> > >> >
> > >> > M: +91 98860 60539 | O: +91 (80) 67935867
> > >> > shanker.balan@shapeblue.com | www.shapeblue.com |
> > >> > Twitter:@shapeblue ShapeBlue Services India LLP, 22nd floor, Unit
> > >> > 2201A, World Trade Centre, Bangalore - 560 055
> > >> >
> > >> > This email and any attachments to it may be confidential and are
> > >> intended
> > >> > solely for the use of the individual to whom it is addressed. Any
> > >> > views
> > >> or
> > >> > opinions expressed are solely those of the author and do not
> > >> > necessarily represent those of Shape Blue Ltd or related companies.
> > >> > If you are not
> > >> the
> > >> > intended recipient of this email, you must neither take any action
> > >> > based upon its contents, nor copy or show it to anyone. Please
> > >> > contact the
> > >> sender if
> > >> > you believe you have received this email in error. Shape Blue Ltd
> > >> > is a company incorporated in England & Wales. ShapeBlue Services
> > >> > India LLP
> > >> is a
> > >> > company incorporated in India and is operated under license from
> > >> > Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company
> > >> > incorporated
> > >> in
> > >> > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue
> > >> > is a registered trademark.
> > >>
> > >>
> > >>
> > >
>
>
RE: Management IP address pool exhausted
Posted by "Musayev, Ilya" <im...@webmd.net>.
Dan,
Ping me on IRC, nick serverchief
Regards
ilya
> -----Original Message-----
> From: Daniel Hertanu [mailto:dart@unixmob.com]
> Sent: Wednesday, September 25, 2013 6:44 AM
> To: users@cloudstack.apache.org
> Subject: Re: Management IP address pool exhausted
>
> Hi
>
> So I re-installed CS, reboot Xen host without entering it first in Maintenace
> Mode, then got the management IP pool fully used. I followed again your
> indications but to no avail.
> In the meantime SSVM has reached number 112 (s-112-VM).
>
> I would really appreciate any other idea.
>
> Thanks,
>
> Daniel
>
>
> On Wed, Sep 25, 2013 at 9:36 AM, Daniel Hertanu <da...@unixmob.com>
> wrote:
>
> > Hello Ilya
> >
> > I submit an issue describing the bug:
> > https://issues.apache.org/jira/browse/CLOUDSTACK-4735 (I picked CS
> > version 4.1.0 as 4.1.1 is reported as non existent, but I've seen this
> > behavior before since version 3).
> >
> > I followed your indications but it didn't work. After changing those
> > records in the database, CS would still give the same error. I revert
> > back to the initial db content, but this time nothing came back
> > (initially the SSVM was still running, it was CPVM that didn't start).
> > After this step I tried some tricks on my own and I ended up with a botched
> environment.
> > So I'll do a fresh install, reboot the host and try your suggestions
> > again.
> >
> > Thanks,
> >
> > Daniel
> >
> >
> > On Tue, Sep 24, 2013 at 10:46 PM, Musayev, Ilya
> <im...@webmd.net>wrote:
> >
> >> I'm not certain if this is hypervisor specific - I've seen the same
> >> issue for vSphere - but I though it was me, I think we may need to
> >> revise the system vm ip space decommission process.
> >>
> >> Shanker and/or Daniel,
> >>
> >> Do you mind creating a ticket in issues.apache.org and write out ways
> >> to reproduce this issue.
> >>
> >> ---
> >>
> >> *NOTE* this is purely from memory and some notes I had - proceed at
> >> your own risk....
> >>
> >>
> >>
> >> In the meantime, lets clean up what you have.
> >>
> >> The goal is to release the IPs that are still marked as reserved and
> >> allocated.
> >>
> >> I'd recommend installing mysql workbench
> >>
> >> There are few tables we need to review:
> >> 1) Stop cloudstack management server
> >> 2) Backup mysql db first.
> >>
> >> Note the id of management network referenced in cloud.networks
> >> Select id from cloud.networks where traffic_type="Management"
> >> In my case, Management network has ID of 205
> >> SELECT * FROM cloud.nics where network_id=205; Alter to
> >> Deallocating the ones that are marked as Reserved.
> >>
> >> Do the same for cloud.user_ip_address SELECT * FROM
> >> cloud.user_ip_address where source_network_id=205; Alter from
> >> Allocated to Free
> >>
> >> Stop any systemvms if running, and alter the state from Running to
> >> Stopped or Expunging in cloud.vm_instance table. This will recreate
> >> the systemvm from scratch.
> >>
> >> I think this should have you covered, if not, revert back the mysql db.
> >> Do mysql dump and grep for IPs that should have been released.
> >>
> >> Let me know,
> >>
> >> Regards
> >> ilya
> >>
> >>
> >>
> >> > -----Original Message-----
> >> > From: Shanker Balan [mailto:shanker.balan@shapeblue.com]
> >> > Sent: Tuesday, September 24, 2013 9:44 AM
> >> > To: <us...@cloudstack.apache.org>
> >> > Subject: Re: Management IP address pool exhausted
> >> >
> >> >
> >> > On 24-Sep-2013, at 6:37 PM, Daniel Hertanu <da...@unixmob.com>
> wrote:
> >> >
> >> > > Hello Ilya
> >> > >
> >> > > I repeated the issue by rebooting the Xen host without having it
> >> > > in maintenance mode in CS. At this moment I have the SSVM up and
> >> > > running and a virtual router (not related to the management IPs, I
> know).
> >> > > Console Proxy VM can't be started because:
> >> > >
> >> > > 2013-09-24 14:56:24,410 INFO
> >> > > [cloud.vm.VirtualMachineManagerImpl]
> >> > > (Job-Executor-22:job-72) Insufficient capacity
> >> > > com.cloud.exception.InsufficientAddressCapacityException: Unable
> >> > > to get a management ip addressScope=interface com.cloud.dc.Pod;
> >> > > id=1
> >> > >
> >> >
> >> >
> >> >
> >> > Hi Ilya,
> >> >
> >> > I have run into the same issue too. It is easily reproducible by
> >> > having
> >> only one
> >> > host in the Zone and rebooting it without enabling maintenance mode
> >> > on
> >> it.
> >> >
> >> > This issue exists for both XenServer and KVM host when I tried.
> >> >
> >> >
> >> > --
> >> > @shankerbalan
> >> >
> >> > M: +91 98860 60539 | O: +91 (80) 67935867
> >> > shanker.balan@shapeblue.com | www.shapeblue.com |
> >> > Twitter:@shapeblue ShapeBlue Services India LLP, 22nd floor, Unit
> >> > 2201A, World Trade Centre, Bangalore - 560 055
> >> >
> >> > This email and any attachments to it may be confidential and are
> >> intended
> >> > solely for the use of the individual to whom it is addressed. Any
> >> > views
> >> or
> >> > opinions expressed are solely those of the author and do not
> >> > necessarily represent those of Shape Blue Ltd or related companies.
> >> > If you are not
> >> the
> >> > intended recipient of this email, you must neither take any action
> >> > based upon its contents, nor copy or show it to anyone. Please
> >> > contact the
> >> sender if
> >> > you believe you have received this email in error. Shape Blue Ltd
> >> > is a company incorporated in England & Wales. ShapeBlue Services
> >> > India LLP
> >> is a
> >> > company incorporated in India and is operated under license from
> >> > Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company
> >> > incorporated
> >> in
> >> > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue
> >> > is a registered trademark.
> >>
> >>
> >>
> >
Re: Management IP address pool exhausted
Posted by Daniel Hertanu <da...@unixmob.com>.
Hi
So I re-installed CS, reboot Xen host without entering it first in
Maintenace Mode, then got the management IP pool fully used. I followed
again your indications but to no avail.
In the meantime SSVM has reached number 112 (s-112-VM).
I would really appreciate any other idea.
Thanks,
Daniel
On Wed, Sep 25, 2013 at 9:36 AM, Daniel Hertanu <da...@unixmob.com> wrote:
> Hello Ilya
>
> I submit an issue describing the bug:
> https://issues.apache.org/jira/browse/CLOUDSTACK-4735 (I picked CS
> version 4.1.0 as 4.1.1 is reported as non existent, but I've seen this
> behavior before since version 3).
>
> I followed your indications but it didn't work. After changing those
> records in the database, CS would still give the same error. I revert back
> to the initial db content, but this time nothing came back (initially the
> SSVM was still running, it was CPVM that didn't start). After this step I
> tried some tricks on my own and I ended up with a botched environment.
> So I'll do a fresh install, reboot the host and try your suggestions
> again.
>
> Thanks,
>
> Daniel
>
>
> On Tue, Sep 24, 2013 at 10:46 PM, Musayev, Ilya <im...@webmd.net>wrote:
>
>> I'm not certain if this is hypervisor specific - I've seen the same issue
>> for vSphere - but I though it was me, I think we may need to revise the
>> system vm ip space decommission process.
>>
>> Shanker and/or Daniel,
>>
>> Do you mind creating a ticket in issues.apache.org and write out ways to
>> reproduce this issue.
>>
>> ---
>>
>> *NOTE* this is purely from memory and some notes I had - proceed at your
>> own risk....
>>
>>
>>
>> In the meantime, lets clean up what you have.
>>
>> The goal is to release the IPs that are still marked as reserved and
>> allocated.
>>
>> I'd recommend installing mysql workbench
>>
>> There are few tables we need to review:
>> 1) Stop cloudstack management server
>> 2) Backup mysql db first.
>>
>> Note the id of management network referenced in cloud.networks
>> Select id from cloud.networks where traffic_type="Management"
>> In my case, Management network has ID of 205
>> SELECT * FROM cloud.nics where network_id=205;
>> Alter to Deallocating the ones that are marked as Reserved.
>>
>> Do the same for cloud.user_ip_address
>> SELECT * FROM cloud.user_ip_address where source_network_id=205;
>> Alter from Allocated to Free
>>
>> Stop any systemvms if running, and alter the state from Running to
>> Stopped or Expunging in cloud.vm_instance table. This will recreate the
>> systemvm from scratch.
>>
>> I think this should have you covered, if not, revert back the mysql db.
>> Do mysql dump and grep for IPs that should have been released.
>>
>> Let me know,
>>
>> Regards
>> ilya
>>
>>
>>
>> > -----Original Message-----
>> > From: Shanker Balan [mailto:shanker.balan@shapeblue.com]
>> > Sent: Tuesday, September 24, 2013 9:44 AM
>> > To: <us...@cloudstack.apache.org>
>> > Subject: Re: Management IP address pool exhausted
>> >
>> >
>> > On 24-Sep-2013, at 6:37 PM, Daniel Hertanu <da...@unixmob.com> wrote:
>> >
>> > > Hello Ilya
>> > >
>> > > I repeated the issue by rebooting the Xen host without having it in
>> > > maintenance mode in CS. At this moment I have the SSVM up and running
>> > > and a virtual router (not related to the management IPs, I know).
>> > > Console Proxy VM can't be started because:
>> > >
>> > > 2013-09-24 14:56:24,410 INFO [cloud.vm.VirtualMachineManagerImpl]
>> > > (Job-Executor-22:job-72) Insufficient capacity
>> > > com.cloud.exception.InsufficientAddressCapacityException: Unable to
>> > > get a management ip addressScope=interface com.cloud.dc.Pod; id=1
>> > >
>> >
>> >
>> >
>> > Hi Ilya,
>> >
>> > I have run into the same issue too. It is easily reproducible by having
>> only one
>> > host in the Zone and rebooting it without enabling maintenance mode on
>> it.
>> >
>> > This issue exists for both XenServer and KVM host when I tried.
>> >
>> >
>> > --
>> > @shankerbalan
>> >
>> > M: +91 98860 60539 | O: +91 (80) 67935867 shanker.balan@shapeblue.com |
>> > www.shapeblue.com | Twitter:@shapeblue ShapeBlue Services India LLP,
>> > 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
>> >
>> > This email and any attachments to it may be confidential and are
>> intended
>> > solely for the use of the individual to whom it is addressed. Any views
>> or
>> > opinions expressed are solely those of the author and do not necessarily
>> > represent those of Shape Blue Ltd or related companies. If you are not
>> the
>> > intended recipient of this email, you must neither take any action based
>> > upon its contents, nor copy or show it to anyone. Please contact the
>> sender if
>> > you believe you have received this email in error. Shape Blue Ltd is a
>> > company incorporated in England & Wales. ShapeBlue Services India LLP
>> is a
>> > company incorporated in India and is operated under license from Shape
>> > Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated
>> in
>> > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a
>> > registered trademark.
>>
>>
>>
>
Re: Management IP address pool exhausted
Posted by Daniel Hertanu <da...@unixmob.com>.
Hello Ilya
I submit an issue describing the bug:
https://issues.apache.org/jira/browse/CLOUDSTACK-4735 (I picked CS version
4.1.0 as 4.1.1 is reported as non existent, but I've seen this behavior
before since version 3).
I followed your indications but it didn't work. After changing those
records in the database, CS would still give the same error. I revert back
to the initial db content, but this time nothing came back (initially the
SSVM was still running, it was CPVM that didn't start). After this step I
tried some tricks on my own and I ended up with a botched environment.
So I'll do a fresh install, reboot the host and try your suggestions again.
Thanks,
Daniel
On Tue, Sep 24, 2013 at 10:46 PM, Musayev, Ilya <im...@webmd.net> wrote:
> I'm not certain if this is hypervisor specific - I've seen the same issue
> for vSphere - but I though it was me, I think we may need to revise the
> system vm ip space decommission process.
>
> Shanker and/or Daniel,
>
> Do you mind creating a ticket in issues.apache.org and write out ways to
> reproduce this issue.
>
> ---
>
> *NOTE* this is purely from memory and some notes I had - proceed at your
> own risk....
>
>
>
> In the meantime, lets clean up what you have.
>
> The goal is to release the IPs that are still marked as reserved and
> allocated.
>
> I'd recommend installing mysql workbench
>
> There are few tables we need to review:
> 1) Stop cloudstack management server
> 2) Backup mysql db first.
>
> Note the id of management network referenced in cloud.networks
> Select id from cloud.networks where traffic_type="Management"
> In my case, Management network has ID of 205
> SELECT * FROM cloud.nics where network_id=205;
> Alter to Deallocating the ones that are marked as Reserved.
>
> Do the same for cloud.user_ip_address
> SELECT * FROM cloud.user_ip_address where source_network_id=205;
> Alter from Allocated to Free
>
> Stop any systemvms if running, and alter the state from Running to Stopped
> or Expunging in cloud.vm_instance table. This will recreate the systemvm
> from scratch.
>
> I think this should have you covered, if not, revert back the mysql db. Do
> mysql dump and grep for IPs that should have been released.
>
> Let me know,
>
> Regards
> ilya
>
>
>
> > -----Original Message-----
> > From: Shanker Balan [mailto:shanker.balan@shapeblue.com]
> > Sent: Tuesday, September 24, 2013 9:44 AM
> > To: <us...@cloudstack.apache.org>
> > Subject: Re: Management IP address pool exhausted
> >
> >
> > On 24-Sep-2013, at 6:37 PM, Daniel Hertanu <da...@unixmob.com> wrote:
> >
> > > Hello Ilya
> > >
> > > I repeated the issue by rebooting the Xen host without having it in
> > > maintenance mode in CS. At this moment I have the SSVM up and running
> > > and a virtual router (not related to the management IPs, I know).
> > > Console Proxy VM can't be started because:
> > >
> > > 2013-09-24 14:56:24,410 INFO [cloud.vm.VirtualMachineManagerImpl]
> > > (Job-Executor-22:job-72) Insufficient capacity
> > > com.cloud.exception.InsufficientAddressCapacityException: Unable to
> > > get a management ip addressScope=interface com.cloud.dc.Pod; id=1
> > >
> >
> >
> >
> > Hi Ilya,
> >
> > I have run into the same issue too. It is easily reproducible by having
> only one
> > host in the Zone and rebooting it without enabling maintenance mode on
> it.
> >
> > This issue exists for both XenServer and KVM host when I tried.
> >
> >
> > --
> > @shankerbalan
> >
> > M: +91 98860 60539 | O: +91 (80) 67935867 shanker.balan@shapeblue.com |
> > www.shapeblue.com | Twitter:@shapeblue ShapeBlue Services India LLP,
> > 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
> >
> > This email and any attachments to it may be confidential and are intended
> > solely for the use of the individual to whom it is addressed. Any views
> or
> > opinions expressed are solely those of the author and do not necessarily
> > represent those of Shape Blue Ltd or related companies. If you are not
> the
> > intended recipient of this email, you must neither take any action based
> > upon its contents, nor copy or show it to anyone. Please contact the
> sender if
> > you believe you have received this email in error. Shape Blue Ltd is a
> > company incorporated in England & Wales. ShapeBlue Services India LLP is
> a
> > company incorporated in India and is operated under license from Shape
> > Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a
> > registered trademark.
>
>
>
RE: Management IP address pool exhausted
Posted by "Musayev, Ilya" <im...@webmd.net>.
I'm not certain if this is hypervisor specific - I've seen the same issue for vSphere - but I though it was me, I think we may need to revise the system vm ip space decommission process.
Shanker and/or Daniel,
Do you mind creating a ticket in issues.apache.org and write out ways to reproduce this issue.
---
*NOTE* this is purely from memory and some notes I had - proceed at your own risk....
In the meantime, lets clean up what you have.
The goal is to release the IPs that are still marked as reserved and allocated.
I'd recommend installing mysql workbench
There are few tables we need to review:
1) Stop cloudstack management server
2) Backup mysql db first.
Note the id of management network referenced in cloud.networks
Select id from cloud.networks where traffic_type="Management"
In my case, Management network has ID of 205
SELECT * FROM cloud.nics where network_id=205;
Alter to Deallocating the ones that are marked as Reserved.
Do the same for cloud.user_ip_address
SELECT * FROM cloud.user_ip_address where source_network_id=205;
Alter from Allocated to Free
Stop any systemvms if running, and alter the state from Running to Stopped or Expunging in cloud.vm_instance table. This will recreate the systemvm from scratch.
I think this should have you covered, if not, revert back the mysql db. Do mysql dump and grep for IPs that should have been released.
Let me know,
Regards
ilya
> -----Original Message-----
> From: Shanker Balan [mailto:shanker.balan@shapeblue.com]
> Sent: Tuesday, September 24, 2013 9:44 AM
> To: <us...@cloudstack.apache.org>
> Subject: Re: Management IP address pool exhausted
>
>
> On 24-Sep-2013, at 6:37 PM, Daniel Hertanu <da...@unixmob.com> wrote:
>
> > Hello Ilya
> >
> > I repeated the issue by rebooting the Xen host without having it in
> > maintenance mode in CS. At this moment I have the SSVM up and running
> > and a virtual router (not related to the management IPs, I know).
> > Console Proxy VM can't be started because:
> >
> > 2013-09-24 14:56:24,410 INFO [cloud.vm.VirtualMachineManagerImpl]
> > (Job-Executor-22:job-72) Insufficient capacity
> > com.cloud.exception.InsufficientAddressCapacityException: Unable to
> > get a management ip addressScope=interface com.cloud.dc.Pod; id=1
> >
>
>
>
> Hi Ilya,
>
> I have run into the same issue too. It is easily reproducible by having only one
> host in the Zone and rebooting it without enabling maintenance mode on it.
>
> This issue exists for both XenServer and KVM host when I tried.
>
>
> --
> @shankerbalan
>
> M: +91 98860 60539 | O: +91 (80) 67935867 shanker.balan@shapeblue.com |
> www.shapeblue.com | Twitter:@shapeblue ShapeBlue Services India LLP,
> 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
>
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender if
> you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape
> Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a
> registered trademark.
Re: Management IP address pool exhausted
Posted by Shanker Balan <sh...@shapeblue.com>.
On 24-Sep-2013, at 6:37 PM, Daniel Hertanu <da...@unixmob.com> wrote:
> Hello Ilya
>
> I repeated the issue by rebooting the Xen host without having it in
> maintenance mode in CS. At this moment I have the SSVM up and running and a
> virtual router (not related to the management IPs, I know). Console Proxy
> VM can't be started because:
>
> 2013-09-24 14:56:24,410 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-22:job-72) Insufficient capacity
> com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
> management ip addressScope=interface com.cloud.dc.Pod; id=1
>
Hi Ilya,
I have run into the same issue too. It is easily reproducible by having
only one host in the Zone and rebooting it without enabling maintenance mode
on it.
This issue exists for both XenServer and KVM host when I tried.
--
@shankerbalan
M: +91 98860 60539 | O: +91 (80) 67935867
shanker.balan@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: Management IP address pool exhausted
Posted by Daniel Hertanu <da...@unixmob.com>.
Hello Ilya
I repeated the issue by rebooting the Xen host without having it in
maintenance mode in CS. At this moment I have the SSVM up and running and a
virtual router (not related to the management IPs, I know). Console Proxy
VM can't be started because:
2013-09-24 14:56:24,410 INFO [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-22:job-72) Insufficient capacity
com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
management ip addressScope=interface com.cloud.dc.Pod; id=1
The queries you were asking about:
mysql> SELECT * FROM cloud.nics where state="Reserved" and
(vm_type="ConsoleProxy" or vm_type="SecondaryStorageVM");
+----+--------------------------------------+-------------+-------------------+----------------+---------------+---------------+---------+-----------------+------------+--------+----------+----------+---------------------------+--------------------------------------+-----------+---------------------+----------------+-------------+-------------+--------------------+---------------------+---------+-------------+----------+
| id | uuid | instance_id | mac_address
| ip4_address | netmask | gateway | ip_type |
broadcast_uri | network_id | mode | state | strategy | reserver_name
| reservation_id | device_id |
update_time | isolation_uri | ip6_address | default_nic | vm_type
| created | removed | ip6_gateway | ip6_cidr |
+----+--------------------------------------+-------------+-------------------+----------------+---------------+---------------+---------+-----------------+------------+--------+----------+----------+---------------------------+--------------------------------------+-----------+---------------------+----------------+-------------+-------------+--------------------+---------------------+---------+-------------+----------+
| 1 | c3c2d45d-94f9-4172-9eee-7829106380de | 1 |
06:e9:ea:00:00:0d | 195.238.173.10 | 255.255.255.0 | 195.238.173.1 | Ip4
| vlan://untagged | 204 | Dhcp | Reserved | Start |
DirectPodBasedNetworkGuru | 6591a371-15e3-4396-b14e-d23de671bdaa |
2 | 2013-09-24 14:54:33 | ec2://untagged | NULL | 1 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL | NULL | NULL
|
| 2 | d2c11105-22fd-4c94-b54b-4039347e3ec2 | 1 |
0e:00:a9:fe:02:ea | 169.254.2.234 | 255.255.0.0 | 169.254.0.1 | Ip4
| NULL | 202 | Static | Reserved | Start |
ControlNetworkGuru | 6591a371-15e3-4396-b14e-d23de671bdaa |
0 | 2013-09-24 14:54:33 | NULL | NULL | 0 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL | NULL | NULL
|
| 3 | 8c1c909c-82fb-4786-b1e6-efc9e3098d51 | 1 |
06:17:8c:00:00:04 | 192.168.0.11 | 255.255.255.0 | 192.168.0.1 | Ip4
| NULL | 201 | Static | Reserved | Start |
PodBasedNetworkGuru | 6591a371-15e3-4396-b14e-d23de671bdaa |
1 | 2013-09-24 14:54:34 | NULL | NULL | 0 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL | NULL | NULL
|
| 4 | 0f440319-18b5-4129-9ff9-37088f794107 | 1 |
06:61:fc:00:00:08 | 192.168.0.15 | 255.255.255.0 | 192.168.0.1 | Ip4
| NULL | 203 | Static | Reserved | Start |
StorageNetworkGuru | 6591a371-15e3-4396-b14e-d23de671bdaa |
3 | 2013-09-24 14:54:34 | NULL | NULL | 0 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL | NULL | NULL
|
+----+--------------------------------------+-------------+-------------------+----------------+---------------+---------------+---------+-----------------+------------+--------+----------+----------+---------------------------+--------------------------------------+-----------+---------------------+----------------+-------------+-------------+--------------------+---------------------+---------+-------------+----------+
mysql> SELECT * FROM cloud.vm_instance where (name like "s-%" or name like
"v-%");
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+------------------+------+------+-------+-------+-----------+--------------+---------------+
| id | name | uuid | instance_name |
state | vm_template_id | guest_os_id | private_mac_address |
private_ip_address | pod_id | data_center_id | host_id | last_host_id |
proxy_id | proxy_assign_time | vnc_password
| ha_enabled | limit_cpu_use | update_count | update_time | created
| removed | type | vm_type |
account_id | domain_id | service_offering_id | reservation_id
| hypervisor_type | disk_offering_id | cpu | ram | owner | speed
| host_name | display_name | desired_state |
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+------------------+------+------+-------+-------+-----------+--------------+---------------+
| 1 | s-1-VM | 2cb5f8db-91eb-4f16-a6fa-4bfed79c51b6 | s-1-VM |
Running | 1 | 133 | 06:17:8c:00:00:04 | 192.168.0.11
| 1 | 1 | 1 | 1 | NULL | NULL
| gGI1kYniavKLLwWWHO1/8oL62JDMUdEbsSpj48XfVCo= | 0 |
0 | 13 | 2013-09-24 13:55:14 | 2013-09-23 12:06:25 |
NULL | SecondaryStorageVm | SecondaryStorageVm | 1 | 1
| 9 | 6591a371-15e3-4396-b14e-d23de671bdaa | XenServer
| NULL | NULL | NULL | NULL | NULL | NULL | NULL
| NULL |
| 2 | v-2-VM | c1975f1c-300f-4a6d-879f-2b70b4842a8f | v-2-VM |
Stopped | 1 | 133 | 06:c4:62:00:00:03 | 192.168.0.10
| 1 | 1 | NULL | 1 | NULL | NULL
| wRV0PYKhMMKR7ENEvJ212Zi3FP//FV1eGmTfIkrkLX4= | 0 |
0 | 371 | 2013-09-24 14:03:35 | 2013-09-23 12:06:26 |
NULL | ConsoleProxy | ConsoleProxy | 1 | 1
| 10 | 1dea74ec-ee0c-4b6c-bc62-43093572f0f1 | XenServer
| NULL | NULL | NULL | NULL | NULL | NULL | NULL
| NULL |
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+------------------+------+------+-------+-------+-----------+--------------+---------------+
The expunge interval and delay is setup to 60 seconds.
Thanks,
Daniel
On Tue, Sep 24, 2013 at 12:34 AM, Musayev, Ilya <im...@webmd.net> wrote:
> I suspect the issue with IPs not being released right away.
>
> What do you get for running this query
>
> SELECT * FROM cloud.nics where state="Reserved" and
> (vm_type="ConsoleProxy" or vm_type="SecondaryStorageVM");
>
> SELECT * FROM cloud.vm_instance where (name like "s-%" or name like "v-%");
>
> If its set to expunging, what is the expunge interval set to in global
> settings?
>
>
> > -----Original Message-----
> > From: Daniel Hertanu [mailto:dart@unixmob.com]
> > Sent: Monday, September 23, 2013 10:08 AM
> > To: users@cloudstack.apache.org
> > Subject: Management IP address pool exhausted
> >
> > Hi
> >
> > I'm running CS 4.1.1 on CentOS 6.4 64 bit with one host Citrix Xenserver
> 6.1.
> >
> > Few days ago I ran into a problem while I was configuring CS. The zone
> was
> > active, Xen host added, SSVM and CPVM up & running when someone from
> > our team rebooted the Xen host for whatever reason. Even though
> > CloudStack was complaining about the missing host in the logs, it
> deleted the
> > old SSVM and tried to deploy another one, it failed, then another one,
> and so
> > on, until the management IP addresses pool was exhausted, moment when
> > CloudStack reported the new error - that there are not enough management
> > IPs. The host came back online later but the situation didn't changed. I
> > lowered the expunge.delay and expunge.interval to 5 minutes, still no
> > change.
> > I wiped out CS installation and I start all over again. I was curious
> about the
> > previous issue, so I manually rebooted the Xen host (without putting it
> in
> > Maintenance mode in CS) and the issue showed up again.
> > While I admit that having one host is not ideal, I have two questions:
> > - why would CS try to deploy any VM when it actually knows there are no
> > valid hosts
> > - how do I make those management IPs available again after CS tried to
> > deploy 10-20 SSVMs and it exhausted the management IPs pool?
> >
> > Thanks,
> >
> > Daniel
>
>
RE: Management IP address pool exhausted
Posted by "Musayev, Ilya" <im...@webmd.net>.
I suspect the issue with IPs not being released right away.
What do you get for running this query
SELECT * FROM cloud.nics where state="Reserved" and (vm_type="ConsoleProxy" or vm_type="SecondaryStorageVM");
SELECT * FROM cloud.vm_instance where (name like "s-%" or name like "v-%");
If its set to expunging, what is the expunge interval set to in global settings?
> -----Original Message-----
> From: Daniel Hertanu [mailto:dart@unixmob.com]
> Sent: Monday, September 23, 2013 10:08 AM
> To: users@cloudstack.apache.org
> Subject: Management IP address pool exhausted
>
> Hi
>
> I'm running CS 4.1.1 on CentOS 6.4 64 bit with one host Citrix Xenserver 6.1.
>
> Few days ago I ran into a problem while I was configuring CS. The zone was
> active, Xen host added, SSVM and CPVM up & running when someone from
> our team rebooted the Xen host for whatever reason. Even though
> CloudStack was complaining about the missing host in the logs, it deleted the
> old SSVM and tried to deploy another one, it failed, then another one, and so
> on, until the management IP addresses pool was exhausted, moment when
> CloudStack reported the new error - that there are not enough management
> IPs. The host came back online later but the situation didn't changed. I
> lowered the expunge.delay and expunge.interval to 5 minutes, still no
> change.
> I wiped out CS installation and I start all over again. I was curious about the
> previous issue, so I manually rebooted the Xen host (without putting it in
> Maintenance mode in CS) and the issue showed up again.
> While I admit that having one host is not ideal, I have two questions:
> - why would CS try to deploy any VM when it actually knows there are no
> valid hosts
> - how do I make those management IPs available again after CS tried to
> deploy 10-20 SSVMs and it exhausted the management IPs pool?
>
> Thanks,
>
> Daniel