You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Martin Emrich <ma...@empolis.com> on 2015/07/07 13:33:09 UTC
Deployment failed on XenServer due to capacity miscalculation
Hi!
I try to create an instance on my ACS (4.4.3, XenServer 6.2 SP1). The deployment fails on the first host in the cluster, as its memory is nearly full.
I see this message:
2015-07-07 13:22:26,713 DEBUG [c.c.c.CapacityManagerImpl] (API-Job-Executor-3:ctx-2496aaad job-44290 ctx-127f9afb FirstFitRoutingAllocator) Free RAM: 4302536704 , Requested RAM: 4294967296
So with the new instance, I would have 7,5MB (!) left... Quite a tight fit, but mathematically acceptable. But XenServer has a different opinion:
2015-07-07 13:22:34,950 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-481:ctx-a44361e8) Task failed! Task record: uuid: ca1797c3-e314-77f3-c4d2-abf12c5dafa2
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Tue Jul 07 13:22:33 CEST 2015
finished: Tue Jul 07 13:22:33 CEST 2015
status: failure
residentOn: com.xensource.xenapi.Host@cf13d8bd
progress: 1.0
type: <none/>
result:
errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 4447010816, 1744826368]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
Of course there's some management overhead necessary, so I won't complain that the VM won't fit. But how can I tell ACS to put in some kind of safety margin in the memory calculations? The host's brother still has 55GB RAM free, so ACS could have deployed the VM there...
Thanks
Martin Emrich
Senior IT Administrator
Empolis Information Management GmbH | Europaallee 10 | 67657 Kaiserslautern | Germany
Phone +49 631 68037-71 | Fax +49 631 68037-77
martin.emrich@empolis.com<mailto:martin.emrich@empolis.com%0d>
www.empolis.com<http://www.empolis.com/>
Sitz Kaiserslautern | Amtsgericht Kaiserslautern HRB 31317
Geschäftsführer: Dr. Stefan Wess, Stefan Volland, Dr. Christian Schulmeyer, Dr. Peter Tepassé
SMART INFORMATION MANAGEMENT
Empolis-Lösungen befähigen Unternehmen und Organisationen, die exponentiell wachsende Menge
strukturierter und unstrukturierter Daten zu analysieren, zu interpretieren und automatisiert zu verarbeiten.
Sie nutzen damit ihr Wissenskapital, um unternehmenskritische Geschäftsprozesse zu optimieren.
Entscheider, Mitarbeiter und Kunden erhalten so stets situations- und aufgabengerecht genau die
Information, die für sie relevant ist.
Abonnieren Sie unseren Newsletter<http://newsletter.empolis.com/art_resource.php?sid=si4n.23ctc3r> | Folgen Sie uns auf Facebook<http://www.facebook.com/EmpolisSoftware> | Besuchen Sie uns auf YouTube<http://www.youtube.com/EmpolisSoftware>
[Signatur.Experton.DE]<http://www.empolis.com/en/news-events/archiv/pressemitteilung-experton.html>
AW: Deployment failed on XenServer due to capacity miscalculation
Posted by Martin Emrich <ma...@empolis.com>.
Hi!
Manually migrating VMs to the free host works (as long as I migrate at most two at once, I guess that the data moving between the local storages produces too much load otherwise).
I also can deploy a VM on a specific host via CloudMonkey, but I'd love to see cloudstack choose a fitting host for me automatically.
Sadly I migrated lots of VMs yesterday to the free host such that each of the old full hosts now has <70% free memory, so for now I cannot reproduce the "cloudstack thinks there's free memory but there isn't" bug.
But can I change the VM placement strategy, so that cloudstack no longer uses the first "free" host, but e.g. always the host _within_ a cluster with the least memory consumption? Just as the DeploymentPlanners are supposed to do _among_different_ clusters?
Then all hosts would be used until the whole cluster is really "full".
Ciao
Martin
-----Ursprüngliche Nachricht-----
Von: Glenn Wagner [mailto:glenn.wagner@shapeblue.com]
Gesendet: Freitag, 28. August 2015 07:48
An: users@cloudstack.apache.org
Betreff: RE: Deployment failed on XenServer due to capacity miscalculation
Hi,
When you migrate the VM's manually to that Host does the process complete?
Just a thought , what you could also do it add a service offering bind to the new host and deploy a VM to test it completes.
Could you post a section of the Management server log file when you deploy the new VM?
Glenn Wagner
Senior Consultant, South Africa
RE: Deployment failed on XenServer due to capacity miscalculation
Posted by Glenn Wagner <gl...@shapeblue.com>.
Hi,
When you migrate the VM's manually to that Host does the process complete?
Just a thought , what you could also do it add a service offering bind to the new host and deploy a VM to test it completes.
Could you post a section of the Management server log file when you deploy the new VM?
Glenn Wagner
Senior Consultant, South Africa
Phone: +27 21 527 0091 | Mobile: +27 73 917 4111
glenn.wagner@shapeblue.com | www.shapeblue.com | Twitter:@shapeBlue
ShapeBlue SA (Pty) Ltd, 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West, Cape Town 7130
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: Thursday, 27 August 2015 10:02 AM
To: users@cloudstack.apache.org
Subject: AW: Deployment failed on XenServer due to capacity miscalculation
Hi!
I still have this problem. We added an additional server to this cluster last week, but it is not being used. ACS tries to start the VM on the first (full) server and aborts, while the new empty server is being ignored.
Now I'll migrate some VMs manually to the new server, but I assume that's not the way it is meant to be...
Ciao
Martin
-----Ursprüngliche Nachricht-----
Von: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Gesendet: Montag, 27. Juli 2015 18:40
An: users@cloudstack.apache.org
Betreff: RE: Deployment failed on XenServer due to capacity miscalculation
This is mostly due to incorrect calculation of XS memory overhead calculation by cloudstack. However, it is expected that the VMs launch is retried on other available host in cluster.
Regards,
Somesh
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: Monday, July 27, 2015 9:55 AM
To: users@cloudstack.apache.org
Subject: AW: Deployment failed on XenServer due to capacity miscalculation
Hi!
(sorry for the delay, was on vacation).
I have never heard of XenServer having this limitation, and have also never experienced it (We use XenServer w/o ACS heavily for several years now, but I never ran into this issue). I also found nothing conclusive... can you provide some documentation on this limitation?
I'll try to evacuate and reboot the host, that might "reset" the stats and help.
Thanks,
Martin
-----Ursprüngliche Nachricht-----
Von: Stephan Seitz [mailto:s.seitz@secretresearchfacility.com]
Gesendet: Sonntag, 12. Juli 2015 22:52
An: users@cloudstack.apache.org
Betreff: Re: Deployment failed on XenServer due to capacity miscalculation
Hi there,
despite not reading the whole thread, I'ld assume that there's simple no single memory segment of the requested size available at your particular xenserver.
Just keep in mind, that Xen partitions memory and - after long run - could not assign a contiguous block, even if the sum of all segemented blocks is greater.
It depends on the version (and if you're brave on different tmem
settings) how reorganization is managed.
Long story short: put the particular host in maintenance, reboot it and get it back into your ACS.
cheers,
Stephan
Am Freitag, den 10.07.2015, 18:02 +0200 schrieb Martin Emrich:
> Hi!
>
> Am 10.07.2015 um 16:42 schrieb Timothy Lothering:
> > Hi Martin,
> >
> > From the logs it seems that ACS has found that the host has sufficient memory capacity, but when it deploys it, it seems there is not enough. It could be a bug whereby technically the system has enough capacity, but during the provisioning stage, it suddenly does not.
> >
> > errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 4447010816, 1744826368]
> >
>
> I read this message as [..., Requested Memory, Available Memory ] on
> the XenServer.
>
> > From the logs it seems you are also using Local Storage (vs Shared), so initially it finds that host 335 has enough memory (albeit ~7MB left) and tries to deploy. The deploy fails and it tries to redeploy the VM using Host 335's storage, which is inaccessible.
> >
> > 1. Have you tried to deploy a 2GB memory Machine on this host?
>
> Yes, won't work either, as the XenServer just had 1,7GB free.
> But I could create a 512MB VM as expected.
>
> Now ACS thinks the host has 3,507 GB free, while XenServer reports
> 1,2GB free. So the gap between what is really free and what ACS thinks
> is free remains the same.
>
> > 2. Do both hosts have the same CPU and memory configuration?
>
> yes, absolutely identical.
>
> > 3. Try to the following:
> >
> > a. Increase the cluster.memory.allocated.capacity.disablethreshold
> > from 0.85 to 0.90 and restart MS - Test redeploy b. Decrease the
> > cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.80
> > and restart MS - Test redeploy
> >
> > The above two tests should get your Host a bit more manoeuvrability and see what happens in the MS Logs.
>
> No effect, as these options refer to a complete cluster, not a single
> host. After changing them, ACS still tries to deploy a new 2GB VM on
> the full host.
>
> I think the key is to somehow force ACS to _ask_ XenServer how much
> memory is really free, instead of doing it's own calculations.
>
> Ciao
>
> Martin
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: Deployment failed on XenServer due to capacity miscalculation
Posted by Prashant s <op...@gmail.com>.
sometimes when i have bad luck spinning vms from the webpage , i try to
force cloudmonkey to create vms for me. it always works.
to create vms using cloudmonkey run the below commands with all ur options
:
*deploy virtualmachine domainid= zoneid= displayname= name= tempalteid=
serviceofferingid= hostid= projectid= ipaddress= *
*cloudmonkey > !for i in {317..326} ; do cloudmonkey deploy virtualmachine
domainid=ca46xxxxxxxx7-11e3-9556-9a21ee1e2575
zoneid=52xxxxxx0dc-b9ce-c907750e0d61 displayname=hadoopvm-p$i
name=hadoop-p$i templateid=0a73a869-3aa6-4d69-94c7-4e9cd9231b73
serviceofferingid=94798ce4-45cc-429b-85f3-b168fadac6bc
networkids=85f3780a-1c26-4f76-9680-26786da626b9 *
*hostid=17b841d3-55c9-4f28-a94e-da20486881a8
**projectid=5dc4f1b8-9c14-47d4-8d0a-6dba9e6d6825
; done*
thanks
prashant
On Thu, Aug 27, 2015 at 12:00 PM, Prashant s <op...@gmail.com> wrote:
> i always use cloudmonkey to spin vm on a particular host
>
> *cloudmonkey > !for i in {317..326} ; do cloudmonkey deploy virtualmachine
> domainid=ca46xxxxxxxx7-11e3-9556-9a21ee1e2575
> zoneid=52xxxxxx0dc-b9ce-c907750e0d61 displayname=hadoopvm-p$i
> name=hadoop-p$i templateid=0a73a869-3aa6-4d69-94c7-4e9cd9231b73
> serviceofferingid=94798ce4-45cc-429b-85f3-b168fadac6bc
> networkids=85f3780a-1c26-4f76-9680-26786da626b9 *
> *hostid=17b841d3-55c9-4f28-a94e-da20486881a8 **projectid=5dc4f1b8-9c14-47d4-8d0a-6dba9e6d6825
> ; done*
>
>
> *cloudmonkey deploy virtualmachine domainid= zoneid= displayname= name=
> tempalteid= serviceofferingid= hostid= projectid= ipaddress= *
>
>
> *thanks *
> *prashant *
>
> On Thu, Aug 27, 2015 at 4:01 AM, Martin Emrich <ma...@empolis.com>
> wrote:
>
>> Hi!
>>
>> I still have this problem. We added an additional server to this cluster
>> last week, but it is not being used. ACS tries to start the VM on the first
>> (full) server and aborts, while the new empty server is being ignored.
>> Now I'll migrate some VMs manually to the new server, but I assume that's
>> not the way it is meant to be...
>>
>> Ciao
>>
>> Martin
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
>> Gesendet: Montag, 27. Juli 2015 18:40
>> An: users@cloudstack.apache.org
>> Betreff: RE: Deployment failed on XenServer due to capacity miscalculation
>>
>> This is mostly due to incorrect calculation of XS memory overhead
>> calculation by cloudstack. However, it is expected that the VMs launch is
>> retried on other available host in cluster.
>>
>> Regards,
>> Somesh
>>
>>
>> -----Original Message-----
>> From: Martin Emrich [mailto:martin.emrich@empolis.com]
>> Sent: Monday, July 27, 2015 9:55 AM
>> To: users@cloudstack.apache.org
>> Subject: AW: Deployment failed on XenServer due to capacity miscalculation
>>
>> Hi!
>>
>> (sorry for the delay, was on vacation).
>>
>> I have never heard of XenServer having this limitation, and have also
>> never experienced it (We use XenServer w/o ACS heavily for several years
>> now, but I never ran into this issue). I also found nothing conclusive...
>> can you provide some documentation on this limitation?
>>
>> I'll try to evacuate and reboot the host, that might "reset" the stats
>> and help.
>>
>> Thanks,
>>
>> Martin
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Stephan Seitz [mailto:s.seitz@secretresearchfacility.com]
>> Gesendet: Sonntag, 12. Juli 2015 22:52
>> An: users@cloudstack.apache.org
>> Betreff: Re: Deployment failed on XenServer due to capacity miscalculation
>>
>> Hi there,
>>
>> despite not reading the whole thread, I'ld assume that there's simple no
>> single memory segment of the requested size available at your particular
>> xenserver.
>> Just keep in mind, that Xen partitions memory and - after long run -
>> could not assign a contiguous block, even if the sum of all segemented
>> blocks is greater.
>> It depends on the version (and if you're brave on different tmem
>> settings) how reorganization is managed.
>> Long story short: put the particular host in maintenance, reboot it and
>> get it back into your ACS.
>>
>> cheers,
>>
>> Stephan
>>
>> Am Freitag, den 10.07.2015, 18:02 +0200 schrieb Martin Emrich:
>> > Hi!
>> >
>> > Am 10.07.2015 um 16:42 schrieb Timothy Lothering:
>> > > Hi Martin,
>> > >
>> > > From the logs it seems that ACS has found that the host has
>> sufficient memory capacity, but when it deploys it, it seems there is not
>> enough. It could be a bug whereby technically the system has enough
>> capacity, but during the provisioning stage, it suddenly does not.
>> > >
>> > > errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 4447010816, 1744826368]
>> > >
>> >
>> > I read this message as [..., Requested Memory, Available Memory ] on
>> > the XenServer.
>> >
>> > > From the logs it seems you are also using Local Storage (vs Shared),
>> so initially it finds that host 335 has enough memory (albeit ~7MB left)
>> and tries to deploy. The deploy fails and it tries to redeploy the VM using
>> Host 335's storage, which is inaccessible.
>> > >
>> > > 1. Have you tried to deploy a 2GB memory Machine on this host?
>> >
>> > Yes, won't work either, as the XenServer just had 1,7GB free.
>> > But I could create a 512MB VM as expected.
>> >
>> > Now ACS thinks the host has 3,507 GB free, while XenServer reports
>> > 1,2GB free. So the gap between what is really free and what ACS thinks
>> > is free remains the same.
>> >
>> > > 2. Do both hosts have the same CPU and memory configuration?
>> >
>> > yes, absolutely identical.
>> >
>> > > 3. Try to the following:
>> > >
>> > > a. Increase the cluster.memory.allocated.capacity.disablethreshold
>> > > from 0.85 to 0.90 and restart MS - Test redeploy b. Decrease the
>> > > cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.80
>> > > and restart MS - Test redeploy
>> > >
>> > > The above two tests should get your Host a bit more manoeuvrability
>> and see what happens in the MS Logs.
>> >
>> > No effect, as these options refer to a complete cluster, not a single
>> > host. After changing them, ACS still tries to deploy a new 2GB VM on
>> > the full host.
>> >
>> > I think the key is to somehow force ACS to _ask_ XenServer how much
>> > memory is really free, instead of doing it's own calculations.
>> >
>> > Ciao
>> >
>> > Martin
>>
>>
>
Re: Deployment failed on XenServer due to capacity miscalculation
Posted by Prashant s <op...@gmail.com>.
i always use cloudmonkey to spin vm on a particular host
*cloudmonkey > !for i in {317..326} ; do cloudmonkey deploy virtualmachine
domainid=ca46xxxxxxxx7-11e3-9556-9a21ee1e2575
zoneid=52xxxxxx0dc-b9ce-c907750e0d61 displayname=hadoopvm-p$i
name=hadoop-p$i templateid=0a73a869-3aa6-4d69-94c7-4e9cd9231b73
serviceofferingid=94798ce4-45cc-429b-85f3-b168fadac6bc
networkids=85f3780a-1c26-4f76-9680-26786da626b9 *
*hostid=17b841d3-55c9-4f28-a94e-da20486881a8
**projectid=5dc4f1b8-9c14-47d4-8d0a-6dba9e6d6825
; done*
*cloudmonkey deploy virtualmachine domainid= zoneid= displayname= name=
tempalteid= serviceofferingid= hostid= projectid= ipaddress= *
*thanks *
*prashant *
On Thu, Aug 27, 2015 at 4:01 AM, Martin Emrich <ma...@empolis.com>
wrote:
> Hi!
>
> I still have this problem. We added an additional server to this cluster
> last week, but it is not being used. ACS tries to start the VM on the first
> (full) server and aborts, while the new empty server is being ignored.
> Now I'll migrate some VMs manually to the new server, but I assume that's
> not the way it is meant to be...
>
> Ciao
>
> Martin
>
> -----Ursprüngliche Nachricht-----
> Von: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Gesendet: Montag, 27. Juli 2015 18:40
> An: users@cloudstack.apache.org
> Betreff: RE: Deployment failed on XenServer due to capacity miscalculation
>
> This is mostly due to incorrect calculation of XS memory overhead
> calculation by cloudstack. However, it is expected that the VMs launch is
> retried on other available host in cluster.
>
> Regards,
> Somesh
>
>
> -----Original Message-----
> From: Martin Emrich [mailto:martin.emrich@empolis.com]
> Sent: Monday, July 27, 2015 9:55 AM
> To: users@cloudstack.apache.org
> Subject: AW: Deployment failed on XenServer due to capacity miscalculation
>
> Hi!
>
> (sorry for the delay, was on vacation).
>
> I have never heard of XenServer having this limitation, and have also
> never experienced it (We use XenServer w/o ACS heavily for several years
> now, but I never ran into this issue). I also found nothing conclusive...
> can you provide some documentation on this limitation?
>
> I'll try to evacuate and reboot the host, that might "reset" the stats and
> help.
>
> Thanks,
>
> Martin
>
>
> -----Ursprüngliche Nachricht-----
> Von: Stephan Seitz [mailto:s.seitz@secretresearchfacility.com]
> Gesendet: Sonntag, 12. Juli 2015 22:52
> An: users@cloudstack.apache.org
> Betreff: Re: Deployment failed on XenServer due to capacity miscalculation
>
> Hi there,
>
> despite not reading the whole thread, I'ld assume that there's simple no
> single memory segment of the requested size available at your particular
> xenserver.
> Just keep in mind, that Xen partitions memory and - after long run - could
> not assign a contiguous block, even if the sum of all segemented blocks is
> greater.
> It depends on the version (and if you're brave on different tmem
> settings) how reorganization is managed.
> Long story short: put the particular host in maintenance, reboot it and
> get it back into your ACS.
>
> cheers,
>
> Stephan
>
> Am Freitag, den 10.07.2015, 18:02 +0200 schrieb Martin Emrich:
> > Hi!
> >
> > Am 10.07.2015 um 16:42 schrieb Timothy Lothering:
> > > Hi Martin,
> > >
> > > From the logs it seems that ACS has found that the host has
> sufficient memory capacity, but when it deploys it, it seems there is not
> enough. It could be a bug whereby technically the system has enough
> capacity, but during the provisioning stage, it suddenly does not.
> > >
> > > errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 4447010816, 1744826368]
> > >
> >
> > I read this message as [..., Requested Memory, Available Memory ] on
> > the XenServer.
> >
> > > From the logs it seems you are also using Local Storage (vs Shared),
> so initially it finds that host 335 has enough memory (albeit ~7MB left)
> and tries to deploy. The deploy fails and it tries to redeploy the VM using
> Host 335's storage, which is inaccessible.
> > >
> > > 1. Have you tried to deploy a 2GB memory Machine on this host?
> >
> > Yes, won't work either, as the XenServer just had 1,7GB free.
> > But I could create a 512MB VM as expected.
> >
> > Now ACS thinks the host has 3,507 GB free, while XenServer reports
> > 1,2GB free. So the gap between what is really free and what ACS thinks
> > is free remains the same.
> >
> > > 2. Do both hosts have the same CPU and memory configuration?
> >
> > yes, absolutely identical.
> >
> > > 3. Try to the following:
> > >
> > > a. Increase the cluster.memory.allocated.capacity.disablethreshold
> > > from 0.85 to 0.90 and restart MS - Test redeploy b. Decrease the
> > > cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.80
> > > and restart MS - Test redeploy
> > >
> > > The above two tests should get your Host a bit more manoeuvrability
> and see what happens in the MS Logs.
> >
> > No effect, as these options refer to a complete cluster, not a single
> > host. After changing them, ACS still tries to deploy a new 2GB VM on
> > the full host.
> >
> > I think the key is to somehow force ACS to _ask_ XenServer how much
> > memory is really free, instead of doing it's own calculations.
> >
> > Ciao
> >
> > Martin
>
>
AW: Deployment failed on XenServer due to capacity miscalculation
Posted by Martin Emrich <ma...@empolis.com>.
Hi!
I still have this problem. We added an additional server to this cluster last week, but it is not being used. ACS tries to start the VM on the first (full) server and aborts, while the new empty server is being ignored.
Now I'll migrate some VMs manually to the new server, but I assume that's not the way it is meant to be...
Ciao
Martin
-----Ursprüngliche Nachricht-----
Von: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Gesendet: Montag, 27. Juli 2015 18:40
An: users@cloudstack.apache.org
Betreff: RE: Deployment failed on XenServer due to capacity miscalculation
This is mostly due to incorrect calculation of XS memory overhead calculation by cloudstack. However, it is expected that the VMs launch is retried on other available host in cluster.
Regards,
Somesh
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: Monday, July 27, 2015 9:55 AM
To: users@cloudstack.apache.org
Subject: AW: Deployment failed on XenServer due to capacity miscalculation
Hi!
(sorry for the delay, was on vacation).
I have never heard of XenServer having this limitation, and have also never experienced it (We use XenServer w/o ACS heavily for several years now, but I never ran into this issue). I also found nothing conclusive... can you provide some documentation on this limitation?
I'll try to evacuate and reboot the host, that might "reset" the stats and help.
Thanks,
Martin
-----Ursprüngliche Nachricht-----
Von: Stephan Seitz [mailto:s.seitz@secretresearchfacility.com]
Gesendet: Sonntag, 12. Juli 2015 22:52
An: users@cloudstack.apache.org
Betreff: Re: Deployment failed on XenServer due to capacity miscalculation
Hi there,
despite not reading the whole thread, I'ld assume that there's simple no single memory segment of the requested size available at your particular xenserver.
Just keep in mind, that Xen partitions memory and - after long run - could not assign a contiguous block, even if the sum of all segemented blocks is greater.
It depends on the version (and if you're brave on different tmem
settings) how reorganization is managed.
Long story short: put the particular host in maintenance, reboot it and get it back into your ACS.
cheers,
Stephan
Am Freitag, den 10.07.2015, 18:02 +0200 schrieb Martin Emrich:
> Hi!
>
> Am 10.07.2015 um 16:42 schrieb Timothy Lothering:
> > Hi Martin,
> >
> > From the logs it seems that ACS has found that the host has sufficient memory capacity, but when it deploys it, it seems there is not enough. It could be a bug whereby technically the system has enough capacity, but during the provisioning stage, it suddenly does not.
> >
> > errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 4447010816, 1744826368]
> >
>
> I read this message as [..., Requested Memory, Available Memory ] on
> the XenServer.
>
> > From the logs it seems you are also using Local Storage (vs Shared), so initially it finds that host 335 has enough memory (albeit ~7MB left) and tries to deploy. The deploy fails and it tries to redeploy the VM using Host 335's storage, which is inaccessible.
> >
> > 1. Have you tried to deploy a 2GB memory Machine on this host?
>
> Yes, won't work either, as the XenServer just had 1,7GB free.
> But I could create a 512MB VM as expected.
>
> Now ACS thinks the host has 3,507 GB free, while XenServer reports
> 1,2GB free. So the gap between what is really free and what ACS thinks
> is free remains the same.
>
> > 2. Do both hosts have the same CPU and memory configuration?
>
> yes, absolutely identical.
>
> > 3. Try to the following:
> >
> > a. Increase the cluster.memory.allocated.capacity.disablethreshold
> > from 0.85 to 0.90 and restart MS - Test redeploy b. Decrease the
> > cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.80
> > and restart MS - Test redeploy
> >
> > The above two tests should get your Host a bit more manoeuvrability and see what happens in the MS Logs.
>
> No effect, as these options refer to a complete cluster, not a single
> host. After changing them, ACS still tries to deploy a new 2GB VM on
> the full host.
>
> I think the key is to somehow force ACS to _ask_ XenServer how much
> memory is really free, instead of doing it's own calculations.
>
> Ciao
>
> Martin
RE: Deployment failed on XenServer due to capacity miscalculation
Posted by Somesh Naidu <So...@citrix.com>.
This is mostly due to incorrect calculation of XS memory overhead calculation by cloudstack. However, it is expected that the VMs launch is retried on other available host in cluster.
Regards,
Somesh
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: Monday, July 27, 2015 9:55 AM
To: users@cloudstack.apache.org
Subject: AW: Deployment failed on XenServer due to capacity miscalculation
Hi!
(sorry for the delay, was on vacation).
I have never heard of XenServer having this limitation, and have also never experienced it (We use XenServer w/o ACS heavily for several years now, but I never ran into this issue). I also found nothing conclusive... can you provide some documentation on this limitation?
I'll try to evacuate and reboot the host, that might "reset" the stats and help.
Thanks,
Martin
-----Ursprüngliche Nachricht-----
Von: Stephan Seitz [mailto:s.seitz@secretresearchfacility.com]
Gesendet: Sonntag, 12. Juli 2015 22:52
An: users@cloudstack.apache.org
Betreff: Re: Deployment failed on XenServer due to capacity miscalculation
Hi there,
despite not reading the whole thread, I'ld assume that there's simple no
single memory segment of the requested size available at your particular
xenserver.
Just keep in mind, that Xen partitions memory and - after long run -
could not assign a contiguous block, even if the sum of all segemented
blocks is greater.
It depends on the version (and if you're brave on different tmem
settings) how reorganization is managed.
Long story short: put the particular host in maintenance, reboot it and
get it back into your ACS.
cheers,
Stephan
Am Freitag, den 10.07.2015, 18:02 +0200 schrieb Martin Emrich:
> Hi!
>
> Am 10.07.2015 um 16:42 schrieb Timothy Lothering:
> > Hi Martin,
> >
> > From the logs it seems that ACS has found that the host has sufficient memory capacity, but when it deploys it, it seems there is not enough. It could be a bug whereby technically the system has enough capacity, but during the provisioning stage, it suddenly does not.
> >
> > errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 4447010816, 1744826368]
> >
>
> I read this message as [..., Requested Memory, Available Memory ] on the
> XenServer.
>
> > From the logs it seems you are also using Local Storage (vs Shared), so initially it finds that host 335 has enough memory (albeit ~7MB left) and tries to deploy. The deploy fails and it tries to redeploy the VM using Host 335's storage, which is inaccessible.
> >
> > 1. Have you tried to deploy a 2GB memory Machine on this host?
>
> Yes, won't work either, as the XenServer just had 1,7GB free.
> But I could create a 512MB VM as expected.
>
> Now ACS thinks the host has 3,507 GB free, while XenServer reports 1,2GB
> free. So the gap between what is really free and what ACS thinks is free
> remains the same.
>
> > 2. Do both hosts have the same CPU and memory configuration?
>
> yes, absolutely identical.
>
> > 3. Try to the following:
> >
> > a. Increase the cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.90 and restart MS - Test redeploy
> > b. Decrease the cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.80 and restart MS - Test redeploy
> >
> > The above two tests should get your Host a bit more manoeuvrability and see what happens in the MS Logs.
>
> No effect, as these options refer to a complete cluster, not a single
> host. After changing them, ACS still tries to deploy a new 2GB VM on the
> full host.
>
> I think the key is to somehow force ACS to _ask_ XenServer how much
> memory is really free, instead of doing it's own calculations.
>
> Ciao
>
> Martin
AW: Deployment failed on XenServer due to capacity miscalculation
Posted by Martin Emrich <ma...@empolis.com>.
Hi!
(sorry for the delay, was on vacation).
I have never heard of XenServer having this limitation, and have also never experienced it (We use XenServer w/o ACS heavily for several years now, but I never ran into this issue). I also found nothing conclusive... can you provide some documentation on this limitation?
I'll try to evacuate and reboot the host, that might "reset" the stats and help.
Thanks,
Martin
-----Ursprüngliche Nachricht-----
Von: Stephan Seitz [mailto:s.seitz@secretresearchfacility.com]
Gesendet: Sonntag, 12. Juli 2015 22:52
An: users@cloudstack.apache.org
Betreff: Re: Deployment failed on XenServer due to capacity miscalculation
Hi there,
despite not reading the whole thread, I'ld assume that there's simple no
single memory segment of the requested size available at your particular
xenserver.
Just keep in mind, that Xen partitions memory and - after long run -
could not assign a contiguous block, even if the sum of all segemented
blocks is greater.
It depends on the version (and if you're brave on different tmem
settings) how reorganization is managed.
Long story short: put the particular host in maintenance, reboot it and
get it back into your ACS.
cheers,
Stephan
Am Freitag, den 10.07.2015, 18:02 +0200 schrieb Martin Emrich:
> Hi!
>
> Am 10.07.2015 um 16:42 schrieb Timothy Lothering:
> > Hi Martin,
> >
> > From the logs it seems that ACS has found that the host has sufficient memory capacity, but when it deploys it, it seems there is not enough. It could be a bug whereby technically the system has enough capacity, but during the provisioning stage, it suddenly does not.
> >
> > errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 4447010816, 1744826368]
> >
>
> I read this message as [..., Requested Memory, Available Memory ] on the
> XenServer.
>
> > From the logs it seems you are also using Local Storage (vs Shared), so initially it finds that host 335 has enough memory (albeit ~7MB left) and tries to deploy. The deploy fails and it tries to redeploy the VM using Host 335's storage, which is inaccessible.
> >
> > 1. Have you tried to deploy a 2GB memory Machine on this host?
>
> Yes, won't work either, as the XenServer just had 1,7GB free.
> But I could create a 512MB VM as expected.
>
> Now ACS thinks the host has 3,507 GB free, while XenServer reports 1,2GB
> free. So the gap between what is really free and what ACS thinks is free
> remains the same.
>
> > 2. Do both hosts have the same CPU and memory configuration?
>
> yes, absolutely identical.
>
> > 3. Try to the following:
> >
> > a. Increase the cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.90 and restart MS - Test redeploy
> > b. Decrease the cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.80 and restart MS - Test redeploy
> >
> > The above two tests should get your Host a bit more manoeuvrability and see what happens in the MS Logs.
>
> No effect, as these options refer to a complete cluster, not a single
> host. After changing them, ACS still tries to deploy a new 2GB VM on the
> full host.
>
> I think the key is to somehow force ACS to _ask_ XenServer how much
> memory is really free, instead of doing it's own calculations.
>
> Ciao
>
> Martin
Re: Deployment failed on XenServer due to capacity miscalculation
Posted by Stephan Seitz <s....@secretresearchfacility.com>.
Hi there,
despite not reading the whole thread, I'ld assume that there's simple no
single memory segment of the requested size available at your particular
xenserver.
Just keep in mind, that Xen partitions memory and - after long run -
could not assign a contiguous block, even if the sum of all segemented
blocks is greater.
It depends on the version (and if you're brave on different tmem
settings) how reorganization is managed.
Long story short: put the particular host in maintenance, reboot it and
get it back into your ACS.
cheers,
Stephan
Am Freitag, den 10.07.2015, 18:02 +0200 schrieb Martin Emrich:
> Hi!
>
> Am 10.07.2015 um 16:42 schrieb Timothy Lothering:
> > Hi Martin,
> >
> > From the logs it seems that ACS has found that the host has sufficient memory capacity, but when it deploys it, it seems there is not enough. It could be a bug whereby technically the system has enough capacity, but during the provisioning stage, it suddenly does not.
> >
> > errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 4447010816, 1744826368]
> >
>
> I read this message as [..., Requested Memory, Available Memory ] on the
> XenServer.
>
> > From the logs it seems you are also using Local Storage (vs Shared), so initially it finds that host 335 has enough memory (albeit ~7MB left) and tries to deploy. The deploy fails and it tries to redeploy the VM using Host 335's storage, which is inaccessible.
> >
> > 1. Have you tried to deploy a 2GB memory Machine on this host?
>
> Yes, won't work either, as the XenServer just had 1,7GB free.
> But I could create a 512MB VM as expected.
>
> Now ACS thinks the host has 3,507 GB free, while XenServer reports 1,2GB
> free. So the gap between what is really free and what ACS thinks is free
> remains the same.
>
> > 2. Do both hosts have the same CPU and memory configuration?
>
> yes, absolutely identical.
>
> > 3. Try to the following:
> >
> > a. Increase the cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.90 and restart MS - Test redeploy
> > b. Decrease the cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.80 and restart MS - Test redeploy
> >
> > The above two tests should get your Host a bit more manoeuvrability and see what happens in the MS Logs.
>
> No effect, as these options refer to a complete cluster, not a single
> host. After changing them, ACS still tries to deploy a new 2GB VM on the
> full host.
>
> I think the key is to somehow force ACS to _ask_ XenServer how much
> memory is really free, instead of doing it's own calculations.
>
> Ciao
>
> Martin
Re: Deployment failed on XenServer due to capacity miscalculation
Posted by Martin Emrich <ma...@empolis.com>.
Hi!
Am 10.07.2015 um 16:42 schrieb Timothy Lothering:
> Hi Martin,
>
> From the logs it seems that ACS has found that the host has sufficient memory capacity, but when it deploys it, it seems there is not enough. It could be a bug whereby technically the system has enough capacity, but during the provisioning stage, it suddenly does not.
>
> errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 4447010816, 1744826368]
>
I read this message as [..., Requested Memory, Available Memory ] on the
XenServer.
> From the logs it seems you are also using Local Storage (vs Shared), so initially it finds that host 335 has enough memory (albeit ~7MB left) and tries to deploy. The deploy fails and it tries to redeploy the VM using Host 335's storage, which is inaccessible.
>
> 1. Have you tried to deploy a 2GB memory Machine on this host?
Yes, won't work either, as the XenServer just had 1,7GB free.
But I could create a 512MB VM as expected.
Now ACS thinks the host has 3,507 GB free, while XenServer reports 1,2GB
free. So the gap between what is really free and what ACS thinks is free
remains the same.
> 2. Do both hosts have the same CPU and memory configuration?
yes, absolutely identical.
> 3. Try to the following:
>
> a. Increase the cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.90 and restart MS - Test redeploy
> b. Decrease the cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.80 and restart MS - Test redeploy
>
> The above two tests should get your Host a bit more manoeuvrability and see what happens in the MS Logs.
No effect, as these options refer to a complete cluster, not a single
host. After changing them, ACS still tries to deploy a new 2GB VM on the
full host.
I think the key is to somehow force ACS to _ask_ XenServer how much
memory is really free, instead of doing it's own calculations.
Ciao
Martin
RE: Deployment failed on XenServer due to capacity miscalculation
Posted by Timothy Lothering <tl...@datacentrix.co.za>.
Hi Martin,
From the logs it seems that ACS has found that the host has sufficient memory capacity, but when it deploys it, it seems there is not enough. It could be a bug whereby technically the system has enough capacity, but during the provisioning stage, it suddenly does not.
errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 4447010816, 1744826368]
From the logs it seems you are also using Local Storage (vs Shared), so initially it finds that host 335 has enough memory (albeit ~7MB left) and tries to deploy. The deploy fails and it tries to redeploy the VM using Host 335's storage, which is inaccessible.
1. Have you tried to deploy a 2GB memory Machine on this host?
2. Do both hosts have the same CPU and memory configuration?
3. Try to the following:
a. Increase the cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.90 and restart MS - Test redeploy
b. Decrease the cluster.memory.allocated.capacity.disablethreshold from 0.85 to 0.80 and restart MS - Test redeploy
The above two tests should get your Host a bit more manoeuvrability and see what happens in the MS Logs.
Kind Regards,
Timothy Lothering
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: 10 July 2015 02:12 PM
To: users@cloudstack.apache.org
Subject: AW: Deployment failed on XenServer due to capacity miscalculation
Yes, here: http://apaste.info/zz4
From clicking on "create" to receiving the deployment error.
Ciao
Martin
-----Ursprüngliche Nachricht-----
Von: Timothy Lothering [mailto:tlothering@datacentrix.co.za]
Gesendet: Freitag, 10. Juli 2015 13:12
An: users@cloudstack.apache.org
Betreff: RE: Deployment failed on XenServer due to capacity miscalculation
Thanks Martin,
Could you please dump your full logs somewhere so we can look into this further?
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: 09 July 2015 03:49 PM
To: users@cloudstack.apache.org
Subject: Re: Deployment failed on XenServer due to capacity miscalculation
Hi!
Am 08.07.2015 um 16:15 schrieb Timothy Lothering:
> Hi Martin,
>
> Thanks for the information.
>
> From the details below, I understand that your Cluster will not allow additional VMs to be created if 85% of the memory is consumed. This value is important as you also need to factor in what amount of memory is required by the Hypervisor (XenServer) to run. Xenserver needs a minimum of 1GB -- Check out the following link: http://support.citrix.com/servlet/KbServlet/download/34970-102-704220/installation.pdf.
Correct. On the other host, only 60% is used, so still enough free space in the cluster. The cluster actually allows new VMs, but tries to start them on the wrong host.
> Secondly, unless you are running memory intensive applications, using a 1:1 ratio is not entirely economically viable, you can safely up the mem.overprovisioning.factor to 2 or 4 (depending on your Memory over provisioning calculations vs CPU). This means that you are essentially doubling or quadrupling up on the actual amount of memory you have available.
>
We prefer to give our users the full memory they request. XenServer does no "real" memory overprovisioning, but proportionally scales VMs down if the hosts real memory is exceeded. Unless there's real dynamic memory scaling (including swapping, deduplication and pressure sensing), we prefer to stay at 1:1.
> Another option to look at is your deployment.planners.order & vm.deployment.planner fields. In my case, I use FirstFitPlanner for both as we prefer to fully utilize each host before provisioning on another host.
That's what I prefer, too.
The problem is not that the algorithm is wrong, but that the data it bases its decisions on is faulty. So not CloudStack fails to create VM, but the underlying XenServer.
Ciao
Martin
Timothy Lothering
Solutions Architect
Managed Services
T: +27877415535
F: +27877415100
C: +27824904099
E: tlothering@datacentrix.co.za
DISCLAIMER NOTICE:
Everything in this e-mail and any attachments relating to the official business of Datacentrix Holdings Ltd. and its subsidiaries
('Datacentrix') is proprietary to Datacentrix. It is confidential, legally privileged and protected by law. Datacentrix does not own and endorse any other content. Views and opinions are those of the sender unless clearly stated as being that of Datacentrix.
The person addressed in the e-mail is the sole authorised recipient. Please notify the sender immediately if it has unintentionally reached you and do not read, disclose or use the content in any way. Datacentrix cannot assure that the integrity of this communication has been maintained nor that it is free of errors, virus, interception or interference.
Timothy Lothering
Solutions Architect
Managed Services
T: +27877415535
F: +27877415100
C: +27824904099
E: tlothering@datacentrix.co.za
DISCLAIMER NOTICE:
Everything in this e-mail and any attachments relating to the official business of Datacentrix Holdings Ltd. and its subsidiaries
('Datacentrix') is proprietary to Datacentrix. It is confidential, legally privileged and protected by law. Datacentrix does not
own and endorse any other content. Views and opinions are those of the sender unless clearly stated as being that of Datacentrix.
The person addressed in the e-mail is the sole authorised recipient. Please notify the sender immediately if it has unintentionally
reached you and do not read, disclose or use the content in any way. Datacentrix cannot assure that the integrity of this communication
has been maintained nor that it is free of errors, virus, interception or interference.
AW: Deployment failed on XenServer due to capacity miscalculation
Posted by Martin Emrich <ma...@empolis.com>.
Yes, here: http://apaste.info/zz4
From clicking on "create" to receiving the deployment error.
Ciao
Martin
-----Ursprüngliche Nachricht-----
Von: Timothy Lothering [mailto:tlothering@datacentrix.co.za]
Gesendet: Freitag, 10. Juli 2015 13:12
An: users@cloudstack.apache.org
Betreff: RE: Deployment failed on XenServer due to capacity miscalculation
Thanks Martin,
Could you please dump your full logs somewhere so we can look into this further?
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: 09 July 2015 03:49 PM
To: users@cloudstack.apache.org
Subject: Re: Deployment failed on XenServer due to capacity miscalculation
Hi!
Am 08.07.2015 um 16:15 schrieb Timothy Lothering:
> Hi Martin,
>
> Thanks for the information.
>
> From the details below, I understand that your Cluster will not allow additional VMs to be created if 85% of the memory is consumed. This value is important as you also need to factor in what amount of memory is required by the Hypervisor (XenServer) to run. Xenserver needs a minimum of 1GB -- Check out the following link: http://support.citrix.com/servlet/KbServlet/download/34970-102-704220/installation.pdf.
Correct. On the other host, only 60% is used, so still enough free space in the cluster. The cluster actually allows new VMs, but tries to start them on the wrong host.
> Secondly, unless you are running memory intensive applications, using a 1:1 ratio is not entirely economically viable, you can safely up the mem.overprovisioning.factor to 2 or 4 (depending on your Memory over provisioning calculations vs CPU). This means that you are essentially doubling or quadrupling up on the actual amount of memory you have available.
>
We prefer to give our users the full memory they request. XenServer does no "real" memory overprovisioning, but proportionally scales VMs down if the hosts real memory is exceeded. Unless there's real dynamic memory scaling (including swapping, deduplication and pressure sensing), we prefer to stay at 1:1.
> Another option to look at is your deployment.planners.order & vm.deployment.planner fields. In my case, I use FirstFitPlanner for both as we prefer to fully utilize each host before provisioning on another host.
That's what I prefer, too.
The problem is not that the algorithm is wrong, but that the data it bases its decisions on is faulty. So not CloudStack fails to create VM, but the underlying XenServer.
Ciao
Martin
Timothy Lothering
Solutions Architect
Managed Services
T: +27877415535
F: +27877415100
C: +27824904099
E: tlothering@datacentrix.co.za
DISCLAIMER NOTICE:
Everything in this e-mail and any attachments relating to the official business of Datacentrix Holdings Ltd. and its subsidiaries
('Datacentrix') is proprietary to Datacentrix. It is confidential, legally privileged and protected by law. Datacentrix does not own and endorse any other content. Views and opinions are those of the sender unless clearly stated as being that of Datacentrix.
The person addressed in the e-mail is the sole authorised recipient. Please notify the sender immediately if it has unintentionally reached you and do not read, disclose or use the content in any way. Datacentrix cannot assure that the integrity of this communication has been maintained nor that it is free of errors, virus, interception or interference.
RE: Deployment failed on XenServer due to capacity miscalculation
Posted by Timothy Lothering <tl...@datacentrix.co.za>.
Thanks Martin,
Could you please dump your full logs somewhere so we can look into this further?
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: 09 July 2015 03:49 PM
To: users@cloudstack.apache.org
Subject: Re: Deployment failed on XenServer due to capacity miscalculation
Hi!
Am 08.07.2015 um 16:15 schrieb Timothy Lothering:
> Hi Martin,
>
> Thanks for the information.
>
> From the details below, I understand that your Cluster will not allow additional VMs to be created if 85% of the memory is consumed. This value is important as you also need to factor in what amount of memory is required by the Hypervisor (XenServer) to run. Xenserver needs a minimum of 1GB -- Check out the following link: http://support.citrix.com/servlet/KbServlet/download/34970-102-704220/installation.pdf.
Correct. On the other host, only 60% is used, so still enough free space in the cluster. The cluster actually allows new VMs, but tries to start them on the wrong host.
> Secondly, unless you are running memory intensive applications, using a 1:1 ratio is not entirely economically viable, you can safely up the mem.overprovisioning.factor to 2 or 4 (depending on your Memory over provisioning calculations vs CPU). This means that you are essentially doubling or quadrupling up on the actual amount of memory you have available.
>
We prefer to give our users the full memory they request. XenServer does no "real" memory overprovisioning, but proportionally scales VMs down if the hosts real memory is exceeded. Unless there's real dynamic memory scaling (including swapping, deduplication and pressure sensing), we prefer to stay at 1:1.
> Another option to look at is your deployment.planners.order & vm.deployment.planner fields. In my case, I use FirstFitPlanner for both as we prefer to fully utilize each host before provisioning on another host.
That's what I prefer, too.
The problem is not that the algorithm is wrong, but that the data it bases its decisions on is faulty. So not CloudStack fails to create VM, but the underlying XenServer.
Ciao
Martin
Timothy Lothering
Solutions Architect
Managed Services
T: +27877415535
F: +27877415100
C: +27824904099
E: tlothering@datacentrix.co.za
DISCLAIMER NOTICE:
Everything in this e-mail and any attachments relating to the official business of Datacentrix Holdings Ltd. and its subsidiaries
('Datacentrix') is proprietary to Datacentrix. It is confidential, legally privileged and protected by law. Datacentrix does not
own and endorse any other content. Views and opinions are those of the sender unless clearly stated as being that of Datacentrix.
The person addressed in the e-mail is the sole authorised recipient. Please notify the sender immediately if it has unintentionally
reached you and do not read, disclose or use the content in any way. Datacentrix cannot assure that the integrity of this communication
has been maintained nor that it is free of errors, virus, interception or interference.
Re: Deployment failed on XenServer due to capacity miscalculation
Posted by Martin Emrich <ma...@empolis.com>.
Hi!
Am 08.07.2015 um 16:15 schrieb Timothy Lothering:
> Hi Martin,
>
> Thanks for the information.
>
> From the details below, I understand that your Cluster will not allow additional VMs to be created if 85% of the memory is consumed. This value is important as you also need to factor in what amount of memory is required by the Hypervisor (XenServer) to run. Xenserver needs a minimum of 1GB -- Check out the following link: http://support.citrix.com/servlet/KbServlet/download/34970-102-704220/installation.pdf.
Correct. On the other host, only 60% is used, so still enough free space
in the cluster. The cluster actually allows new VMs, but tries to start
them on the wrong host.
> Secondly, unless you are running memory intensive applications, using a 1:1 ratio is not entirely economically viable, you can safely up the mem.overprovisioning.factor to 2 or 4 (depending on your Memory over provisioning calculations vs CPU). This means that you are essentially doubling or quadrupling up on the actual amount of memory you have available.
>
We prefer to give our users the full memory they request. XenServer does
no "real" memory overprovisioning, but proportionally scales VMs down if
the hosts real memory is exceeded. Unless there's real dynamic memory
scaling (including swapping, deduplication and pressure sensing), we
prefer to stay at 1:1.
> Another option to look at is your deployment.planners.order & vm.deployment.planner fields. In my case, I use FirstFitPlanner for both as we prefer to fully utilize each host before provisioning on another host.
That's what I prefer, too.
The problem is not that the algorithm is wrong, but that the data it
bases its decisions on is faulty. So not CloudStack fails to create VM,
but the underlying XenServer.
Ciao
Martin
RE: Deployment failed on XenServer due to capacity miscalculation
Posted by Timothy Lothering <tl...@datacentrix.co.za>.
Hi Martin,
Thanks for the information.
From the details below, I understand that your Cluster will not allow additional VMs to be created if 85% of the memory is consumed. This value is important as you also need to factor in what amount of memory is required by the Hypervisor (XenServer) to run. Xenserver needs a minimum of 1GB -- Check out the following link: http://support.citrix.com/servlet/KbServlet/download/34970-102-704220/installation.pdf.
Secondly, unless you are running memory intensive applications, using a 1:1 ratio is not entirely economically viable, you can safely up the mem.overprovisioning.factor to 2 or 4 (depending on your Memory over provisioning calculations vs CPU). This means that you are essentially doubling or quadrupling up on the actual amount of memory you have available.
Another option to look at is your deployment.planners.order & vm.deployment.planner fields. In my case, I use FirstFitPlanner for both as we prefer to fully utilize each host before provisioning on another host.
Kind Regards,
Timothy Lothering
Timothy Lothering
Solutions Architect
Managed Services
T: +27877415535
F: +27877415100
C: +27824904099
E: tlothering@datacentrix.co.za
DISCLAIMER NOTICE:
Everything in this e-mail and any attachments relating to the official business of Datacentrix Holdings Ltd. and its subsidiaries
('Datacentrix') is proprietary to Datacentrix. It is confidential, legally privileged and protected by law. Datacentrix does not
own and endorse any other content. Views and opinions are those of the sender unless clearly stated as being that of Datacentrix.
The person addressed in the e-mail is the sole authorised recipient. Please notify the sender immediately if it has unintentionally
reached you and do not read, disclose or use the content in any way. Datacentrix cannot assure that the integrity of this communication
has been maintained nor that it is free of errors, virus, interception or interference.
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: 08 July 2015 01:45 PM
To: users@cloudstack.apache.org
Subject: AW: Deployment failed on XenServer due to capacity miscalculation
Hi!
The config options are on their default value:
mem.overprovisioning.factor = 1
cluster.memory.allocated.capacity.disablethreshold = 0.85 (Would affect the whole cluster capacity, not that of a single host)
So nothing strange here.
I un- and remanaged the cluster, but no change in numbers :( Also there were no interesting log messages...
Thanks
Martin
-----Ursprüngliche Nachricht-----
Von: Timothy Lothering [mailto:tlothering@datacentrix.co.za]
Gesendet: Mittwoch, 8. Juli 2015 12:34
An: users@cloudstack.apache.org
Betreff: RE: Deployment failed on XenServer due to capacity miscalculation
Hi Martin,
I have seen this before with our CXS environment too.
Could of things to check:
1. What is the mem.overprovisioning.factor field set to?
2. What is the cluster.memory.allocated.capacity.disablethreshold field set to?
Also, try unmanage the XS Cluster, wait a couple of minutes and re-manage it.
Timothy Lothering
Solutions Architect
Managed Services
T: +27877415535
F: +27877415100
C: +27824904099
E: tlothering@datacentrix.co.za
DISCLAIMER NOTICE:
Everything in this e-mail and any attachments relating to the official business of Datacentrix Holdings Ltd. and its subsidiaries
('Datacentrix') is proprietary to Datacentrix. It is confidential, legally privileged and protected by law. Datacentrix does not own and endorse any other content. Views and opinions are those of the sender unless clearly stated as being that of Datacentrix.
The person addressed in the e-mail is the sole authorised recipient. Please notify the sender immediately if it has unintentionally reached you and do not read, disclose or use the content in any way. Datacentrix cannot assure that the integrity of this communication has been maintained nor that it is free of errors, virus, interception or interference.
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: 08 July 2015 12:17 PM
To: users@cloudstack.apache.org
Subject: AW: Deployment failed on XenServer due to capacity miscalculation
Hi!
I tried that ("force reconnect"). But no improvement:
Via cloudmonkey, memorytotal-memoryallocated == 4302535168, that's 4103MB. Via XenServer, the host has only 1667 MB free memory.
Restarting the management server does not solve it, either. I see this:
2015-07-08 12:03:40,612 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) Found 74 VMs on host 335
2015-07-08 12:03:40,641 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) Found 2 VM, not running on host 335
2015-07-08 12:03:40,643 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) No need to calibrate cpu capacity, host:335 usedCpu: 226500 reservedCpu: 0
2015-07-08 12:03:40,643 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) No need to calibrate memory capacity, host:335 usedMem: 124528885760 reservedMem: 0
My current "workaround" is to disable the host in ACS, then the deployment planner takes the other host with enough free memory, but that's not a permanent solution.
Ciao
Martin
-----Ursprüngliche Nachricht-----
Von: Nick Brody [mailto:nickbrody2014@gmail.com]
Gesendet: Mittwoch, 8. Juli 2015 12:05
An: users@cloudstack.apache.org
Betreff: Re: Deployment failed on XenServer due to capacity miscalculation
hi martin, Make sure that you reconnect server
AW: Deployment failed on XenServer due to capacity miscalculation
Posted by Martin Emrich <ma...@empolis.com>.
Hi!
The config options are on their default value:
mem.overprovisioning.factor = 1
cluster.memory.allocated.capacity.disablethreshold = 0.85 (Would affect the whole cluster capacity, not that of a single host)
So nothing strange here.
I un- and remanaged the cluster, but no change in numbers :(
Also there were no interesting log messages...
Thanks
Martin
-----Ursprüngliche Nachricht-----
Von: Timothy Lothering [mailto:tlothering@datacentrix.co.za]
Gesendet: Mittwoch, 8. Juli 2015 12:34
An: users@cloudstack.apache.org
Betreff: RE: Deployment failed on XenServer due to capacity miscalculation
Hi Martin,
I have seen this before with our CXS environment too.
Could of things to check:
1. What is the mem.overprovisioning.factor field set to?
2. What is the cluster.memory.allocated.capacity.disablethreshold field set to?
Also, try unmanage the XS Cluster, wait a couple of minutes and re-manage it.
Timothy Lothering
Solutions Architect
Managed Services
T: +27877415535
F: +27877415100
C: +27824904099
E: tlothering@datacentrix.co.za
DISCLAIMER NOTICE:
Everything in this e-mail and any attachments relating to the official business of Datacentrix Holdings Ltd. and its subsidiaries
('Datacentrix') is proprietary to Datacentrix. It is confidential, legally privileged and protected by law. Datacentrix does not own and endorse any other content. Views and opinions are those of the sender unless clearly stated as being that of Datacentrix.
The person addressed in the e-mail is the sole authorised recipient. Please notify the sender immediately if it has unintentionally reached you and do not read, disclose or use the content in any way. Datacentrix cannot assure that the integrity of this communication has been maintained nor that it is free of errors, virus, interception or interference.
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: 08 July 2015 12:17 PM
To: users@cloudstack.apache.org
Subject: AW: Deployment failed on XenServer due to capacity miscalculation
Hi!
I tried that ("force reconnect"). But no improvement:
Via cloudmonkey, memorytotal-memoryallocated == 4302535168, that's 4103MB. Via XenServer, the host has only 1667 MB free memory.
Restarting the management server does not solve it, either. I see this:
2015-07-08 12:03:40,612 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) Found 74 VMs on host 335
2015-07-08 12:03:40,641 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) Found 2 VM, not running on host 335
2015-07-08 12:03:40,643 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) No need to calibrate cpu capacity, host:335 usedCpu: 226500 reservedCpu: 0
2015-07-08 12:03:40,643 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) No need to calibrate memory capacity, host:335 usedMem: 124528885760 reservedMem: 0
My current "workaround" is to disable the host in ACS, then the deployment planner takes the other host with enough free memory, but that's not a permanent solution.
Ciao
Martin
-----Ursprüngliche Nachricht-----
Von: Nick Brody [mailto:nickbrody2014@gmail.com]
Gesendet: Mittwoch, 8. Juli 2015 12:05
An: users@cloudstack.apache.org
Betreff: Re: Deployment failed on XenServer due to capacity miscalculation
hi martin, Make sure that you reconnect server
RE: Deployment failed on XenServer due to capacity miscalculation
Posted by Timothy Lothering <tl...@datacentrix.co.za>.
Hi Martin,
I have seen this before with our CXS environment too.
Could of things to check:
1. What is the mem.overprovisioning.factor field set to?
2. What is the cluster.memory.allocated.capacity.disablethreshold field set to?
Also, try unmanage the XS Cluster, wait a couple of minutes and re-manage it.
Timothy Lothering
Solutions Architect
Managed Services
T: +27877415535
F: +27877415100
C: +27824904099
E: tlothering@datacentrix.co.za
DISCLAIMER NOTICE:
Everything in this e-mail and any attachments relating to the official business of Datacentrix Holdings Ltd. and its subsidiaries
('Datacentrix') is proprietary to Datacentrix. It is confidential, legally privileged and protected by law. Datacentrix does not
own and endorse any other content. Views and opinions are those of the sender unless clearly stated as being that of Datacentrix.
The person addressed in the e-mail is the sole authorised recipient. Please notify the sender immediately if it has unintentionally
reached you and do not read, disclose or use the content in any way. Datacentrix cannot assure that the integrity of this communication
has been maintained nor that it is free of errors, virus, interception or interference.
-----Original Message-----
From: Martin Emrich [mailto:martin.emrich@empolis.com]
Sent: 08 July 2015 12:17 PM
To: users@cloudstack.apache.org
Subject: AW: Deployment failed on XenServer due to capacity miscalculation
Hi!
I tried that ("force reconnect"). But no improvement:
Via cloudmonkey, memorytotal-memoryallocated == 4302535168, that's 4103MB. Via XenServer, the host has only 1667 MB free memory.
Restarting the management server does not solve it, either. I see this:
2015-07-08 12:03:40,612 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) Found 74 VMs on host 335
2015-07-08 12:03:40,641 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) Found 2 VM, not running on host 335
2015-07-08 12:03:40,643 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) No need to calibrate cpu capacity, host:335 usedCpu: 226500 reservedCpu: 0
2015-07-08 12:03:40,643 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) No need to calibrate memory capacity, host:335 usedMem: 124528885760 reservedMem: 0
My current "workaround" is to disable the host in ACS, then the deployment planner takes the other host with enough free memory, but that's not a permanent solution.
Ciao
Martin
-----Ursprüngliche Nachricht-----
Von: Nick Brody [mailto:nickbrody2014@gmail.com]
Gesendet: Mittwoch, 8. Juli 2015 12:05
An: users@cloudstack.apache.org
Betreff: Re: Deployment failed on XenServer due to capacity miscalculation
hi martin, Make sure that you reconnect server
AW: Deployment failed on XenServer due to capacity miscalculation
Posted by Martin Emrich <ma...@empolis.com>.
Hi!
I tried that ("force reconnect"). But no improvement:
Via cloudmonkey, memorytotal-memoryallocated == 4302535168, that's 4103MB. Via XenServer, the host has only 1667 MB free memory.
Restarting the management server does not solve it, either. I see this:
2015-07-08 12:03:40,612 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) Found 74 VMs on host 335
2015-07-08 12:03:40,641 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) Found 2 VM, not running on host 335
2015-07-08 12:03:40,643 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) No need to calibrate cpu capacity, host:335 usedCpu: 226500 reservedCpu: 0
2015-07-08 12:03:40,643 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-6c463573) No need to calibrate memory capacity, host:335 usedMem: 124528885760 reservedMem: 0
My current "workaround" is to disable the host in ACS, then the deployment planner takes the other host with enough free memory, but that's not a permanent solution.
Ciao
Martin
-----Ursprüngliche Nachricht-----
Von: Nick Brody [mailto:nickbrody2014@gmail.com]
Gesendet: Mittwoch, 8. Juli 2015 12:05
An: users@cloudstack.apache.org
Betreff: Re: Deployment failed on XenServer due to capacity miscalculation
hi martin, Make sure that you reconnect server
Re: Deployment failed on XenServer due to capacity miscalculation
Posted by Nick Brody <ni...@gmail.com>.
hi martin, Make sure that you reconnect server
On Tue, Jul 7, 2015 at 4:33 AM, Martin Emrich <ma...@empolis.com>
wrote:
> Hi!
>
>
>
> I try to create an instance on my ACS (4.4.3, XenServer 6.2 SP1). The
> deployment fails on the first host in the cluster, as its memory is nearly
> full.
>
>
>
> I see this message:
>
>
>
> 2015-07-07 13:22:26,713 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-3:ctx-2496aaad job-44290 ctx-127f9afb
> FirstFitRoutingAllocator) Free RAM: 4302536704 , Requested RAM: 4294967296
>
>
>
> So with the new instance, I would have 7,5MB (!) left… Quite a tight fit,
> but mathematically acceptable. But XenServer has a different opinion:
>
>
>
> 2015-07-07 13:22:34,950 WARN [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-481:ctx-a44361e8) Task failed! Task record:
> uuid: ca1797c3-e314-77f3-c4d2-abf12c5dafa2
>
> nameLabel: Async.VM.start_on
>
> nameDescription:
>
> allowedOperations: []
>
> currentOperations: {}
>
> created: Tue Jul 07 13:22:33 CEST 2015
>
> finished: Tue Jul 07 13:22:33 CEST 2015
>
> status: failure
>
> residentOn: com.xensource.xenapi.Host@cf13d8bd
>
> progress: 1.0
>
> type: <none/>
>
> result:
>
> errorInfo: [HOST_NOT_ENOUGH_FREE_MEMORY, 4447010816, 1744826368]
>
> otherConfig: {}
>
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>
> subtasks: []
>
>
>
>
>
> Of course there’s some management overhead necessary, so I won’t complain
> that the VM won’t fit. But how can I tell ACS to put in some kind of safety
> margin in the memory calculations? The host’s brother still has 55GB RAM
> free, so ACS could have deployed the VM there…
>
>
>
> Thanks
>
>
>
> Martin Emrich
>
> Senior IT Administrator
>
>
>
> Empolis Information Management GmbH | Europaallee 10 | 67657
> Kaiserslautern | Germany
>
> Phone +49 631 68037-71 | Fax +49 631 68037-77
>
> *martin.emrich@empolis.com <martin.emrich@empolis.com%0d>*
>
>
>
> *www.empolis.com <http://www.empolis.com/>*
>
> Sitz Kaiserslautern | Amtsgericht Kaiserslautern HRB 31317
>
> Geschäftsführer: Dr. Stefan Wess, Stefan Volland, Dr. Christian
> Schulmeyer, Dr. Peter Tepassé
>
>
>
> *SMART INFORMATION MANAGEMENT*
> Empolis-Lösungen befähigen Unternehmen und Organisationen, die
> exponentiell wachsende Menge
> strukturierter und unstrukturierter Daten zu analysieren, zu
> interpretieren und automatisiert zu verarbeiten.
>
> Sie nutzen damit ihr Wissenskapital, um unternehmenskritische
> Geschäftsprozesse zu optimieren.
>
> Entscheider, Mitarbeiter und Kunden erhalten so stets situations- und
> aufgabengerecht genau die
>
> Information, die für sie relevant ist.
>
> Abonnieren Sie unseren Newsletter
> <http://newsletter.empolis.com/art_resource.php?sid=si4n.23ctc3r> | Folgen
> Sie uns auf Facebook <http://www.facebook.com/EmpolisSoftware> | Besuchen
> Sie uns auf YouTube <http://www.youtube.com/EmpolisSoftware>
>
> [image: Signatur.Experton.DE]
> <http://www.empolis.com/en/news-events/archiv/pressemitteilung-experton.html>
>
>
>