You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Dmitry Berezhnoy <d....@emzior.ru> on 2019/07/24 15:34:21 UTC
Failed to get free memory without no reason
Hello,
Asynchronous creating VM bring to "Unable to start VM on Host[-1-Routing]
due to failed to get free memory". After that I see ERROR:
InsufficientServerCapacityException.
In previous messages I see large amount of free resources:
2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
(logid:b6e33d80) Current Used CPU: 115708 , Free CPU:76356 ,Requested CPU:
512
2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
(logid:b6e33d80) Current Used RAM: 122003914752 , Free RAM:145667670016
,Requested RAM: 536870912
2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
(logid:b6e33d80) CPU STATS after allocation: for host: 1, old used: 115708,
old reserved: 1536, actual total: 193600, total with overprovisioning:
193600; new used:116220, reserved:1536; requested
cpu:512,alloc_from_last:false
2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
(logid:b6e33d80) RAM STATS after allocation: for host: 1, old used:
122003914752, old reserved: 1610612736, total: 269282197504; new used:
122540785664, reserved: 1610612736; requested mem:
536870912,alloc_from_last:false
How it possible?
Thanks in advance for the help,
Dmitry.
Re: Failed to get free memory without no reason
Posted by Andrija Panic <an...@gmail.com>.
Thx for the bug report!
On Fri, 26 Jul 2019, 14:53 Dmitry Berezhnoy, <d....@emzior.ru> wrote:
> I'm not a Java developer too, but I'm Python developer, so I rewrote code
> of getFreeMemory() to Python and found error in Java code.
> When I create a new VM we take a list of all VMs from libvirt via
> getFreeMemory(). After that we iterate by this list. If amount of VMs so
> large(or libvirt response longer than always) this may take some time. At
> that time one or more VM, which has "Running" state can start stopping by
> another process(job) and not displayed in libvirt anymore. So it bring to
> exception with "libvirt.libvirtError: Domain not found: no domain with
> matching id 157124." message and we move on catch with wrong error
> message(Failed to get free memory).
> I wrote a bug report about this problem
> https://github.com/apache/cloudstack/issues/3526.
>
> пт, 26 июл. 2019 г. в 00:09, Andrija Panic <an...@gmail.com>:
>
> > Based on this (I'm no developer...)
> >
> >
> https://github.com/apache/cloudstack/blob/4.11/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L69
> >
> > ...it seems like CloudStack could not fetch free mem from libvirt - i.e.
> I
> > would check libvirt logs (you can also try to spin a VM manually via
> virsh
> > or Virtual Machine Managed) - or simply restart libvirt daemon.
> >
> > Could you make a conclusion if this error always happens on some
> particular
> > host(s) - libvirt is known to get stuck on very rare occasions - so a
> > restart would help.
> >
> > BTW - you seems to be reaching 1.000.000 VMs created in total ??? If so -
> > that's a nice number :)
> >
> >
> > Cheers
> > Andrija
> >
> >
> >
> >
> > On Thu, 25 Jul 2019 at 17:10, Dmitry Berezhnoy <d....@emzior.ru>
> > wrote:
> >
> > > *4.11.2.0
> > >
> > > чт, 25 июл. 2019 г. в 18:09, Dmitry Berezhnoy <d....@emzior.ru>:
> > >
> > > > First log messages from 10.11.2.0
> > > >
> > > > чт, 25 июл. 2019 г. в 17:56, Andrija Panic <andrija.panic@gmail.com
> >:
> > > >
> > > >> FYI,4.10 is so broken that it's not considered an official releases
> > > (check
> > > >> yourself on the GitHub) - please upgrade to 4.11.3 if possible.
> > > >>
> > > >> Will check logs later today.
> > > >>
> > > >> Andrija
> > > >>
> > > >> On Thu, 25 Jul 2019 at 16:48, Dmitry Berezhnoy <
> d.berezhnoy@emzior.ru
> > >
> > > >> wrote:
> > > >>
> > > >> > Log from that CloudStack station really huge, so I download log
> from
> > > >> > another CloudStack(version 4.10.0) with the same errors.
> > > logid:f0c4abda
> > > >> for
> > > >> > example.
> > > >> >
> > > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > > >> > FirstFitRoutingAllocator) (logid:f0c4abda) Free RAM: 88188272640 ,
> > > >> > Requested RAM: 402653184
> > > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > > >> > FirstFitRoutingAllocator) (logid:f0c4abda) Host has enough CPU and
> > RAM
> > > >> > available
> > > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > > >> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc CPU
> from
> > > >> host:
> > > >> > 1, used: 48556, reserved: 0, actual total: 144000, total with
> > > >> > overprovisioning: 144000; requested
> > > cpu:384,alloc_from_last_host?:false
> > > >> > ,considerReservedCapacity?: true
> > > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > > >> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc MEM
> from
> > > >> host:
> > > >> > 1, used: 45902462976, reserved: 0, total: 134090735616; requested
> > mem:
> > > >> > 402653184,alloc_from_last_host?:false ,considerReservedCapacity?:
> > true
> > > >> > ...
> > > >> > 2019-07-25 15:58:07,660 INFO [c.c.v.VirtualMachineManagerImpl]
> > > >> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659
> > > ctx-fe95df77)
> > > >> > (logid:f0c4abda) Unable to start VM on Host[-1-Routing] due to
> > failed
> > > to
> > > >> > get free memory
> > > >> > ...
> > > >> > 2019-07-25 15:58:14,421 ERROR [c.c.v.VmWorkJobHandlerProxy]
> > > >> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659
> > > ctx-fe95df77)
> > > >> > (logid:f0c4abda) Invocation exception, caused by:
> > > >> > com.cloud.exception.InsufficientServerCapacityException: Unable to
> > > >> create a
> > > >> > deployment for VM[User|i-2-977270-VM]Scope=interface
> > > >> > com.cloud.dc.DataCenter; id=3
> > > >> >
> > > >> > management-server.log https://yadi.sk/d/wzyZVeTKS2kkDQ
> > > >> >
> > > >> > чт, 25 июл. 2019 г. в 16:51, Andrija Panic <
> andrija.panic@gmail.com
> > >:
> > > >> >
> > > >> > > Can you share the whole log file (pastebin.org please, or
> > similar)
> > > ?
> > > >> > >
> > > >> > > On Wed, 24 Jul 2019 at 17:34, Dmitry Berezhnoy <
> > > d.berezhnoy@emzior.ru
> > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hello,
> > > >> > > >
> > > >> > > > Asynchronous creating VM bring to "Unable to start VM on
> > > >> > Host[-1-Routing]
> > > >> > > > due to failed to get free memory". After that I see ERROR:
> > > >> > > > InsufficientServerCapacityException.
> > > >> > > > In previous messages I see large amount of free resources:
> > > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > > >> > ctx-52b48ca6)
> > > >> > > > (logid:b6e33d80) Current Used CPU: 115708 , Free CPU:76356
> > > >> ,Requested
> > > >> > > CPU:
> > > >> > > > 512
> > > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > > >> > ctx-52b48ca6)
> > > >> > > > (logid:b6e33d80) Current Used RAM: 122003914752 , Free
> > > >> RAM:145667670016
> > > >> > > > ,Requested RAM: 536870912
> > > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > > >> > ctx-52b48ca6)
> > > >> > > > (logid:b6e33d80) CPU STATS after allocation: for host: 1, old
> > > used:
> > > >> > > 115708,
> > > >> > > > old reserved: 1536, actual total: 193600, total with
> > > >> overprovisioning:
> > > >> > > > 193600; new used:116220, reserved:1536; requested
> > > >> > > > cpu:512,alloc_from_last:false
> > > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > > >> > ctx-52b48ca6)
> > > >> > > > (logid:b6e33d80) RAM STATS after allocation: for host: 1, old
> > > used:
> > > >> > > > 122003914752, old reserved: 1610612736, total: 269282197504;
> new
> > > >> used:
> > > >> > > > 122540785664, reserved: 1610612736; requested mem:
> > > >> > > > 536870912,alloc_from_last:false
> > > >> > > >
> > > >> > > > How it possible?
> > > >> > > >
> > > >> > > > Thanks in advance for the help,
> > > >> > > > Dmitry.
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > >
> > > >> > > Andrija Panić
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Andrija Panić
> > > >>
> > > >
> > >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
Re: Failed to get free memory without no reason
Posted by Dmitry Berezhnoy <d....@emzior.ru>.
I'm not a Java developer too, but I'm Python developer, so I rewrote code
of getFreeMemory() to Python and found error in Java code.
When I create a new VM we take a list of all VMs from libvirt via
getFreeMemory(). After that we iterate by this list. If amount of VMs so
large(or libvirt response longer than always) this may take some time. At
that time one or more VM, which has "Running" state can start stopping by
another process(job) and not displayed in libvirt anymore. So it bring to
exception with "libvirt.libvirtError: Domain not found: no domain with
matching id 157124." message and we move on catch with wrong error
message(Failed to get free memory).
I wrote a bug report about this problem
https://github.com/apache/cloudstack/issues/3526.
пт, 26 июл. 2019 г. в 00:09, Andrija Panic <an...@gmail.com>:
> Based on this (I'm no developer...)
>
> https://github.com/apache/cloudstack/blob/4.11/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L69
>
> ...it seems like CloudStack could not fetch free mem from libvirt - i.e. I
> would check libvirt logs (you can also try to spin a VM manually via virsh
> or Virtual Machine Managed) - or simply restart libvirt daemon.
>
> Could you make a conclusion if this error always happens on some particular
> host(s) - libvirt is known to get stuck on very rare occasions - so a
> restart would help.
>
> BTW - you seems to be reaching 1.000.000 VMs created in total ??? If so -
> that's a nice number :)
>
>
> Cheers
> Andrija
>
>
>
>
> On Thu, 25 Jul 2019 at 17:10, Dmitry Berezhnoy <d....@emzior.ru>
> wrote:
>
> > *4.11.2.0
> >
> > чт, 25 июл. 2019 г. в 18:09, Dmitry Berezhnoy <d....@emzior.ru>:
> >
> > > First log messages from 10.11.2.0
> > >
> > > чт, 25 июл. 2019 г. в 17:56, Andrija Panic <an...@gmail.com>:
> > >
> > >> FYI,4.10 is so broken that it's not considered an official releases
> > (check
> > >> yourself on the GitHub) - please upgrade to 4.11.3 if possible.
> > >>
> > >> Will check logs later today.
> > >>
> > >> Andrija
> > >>
> > >> On Thu, 25 Jul 2019 at 16:48, Dmitry Berezhnoy <d.berezhnoy@emzior.ru
> >
> > >> wrote:
> > >>
> > >> > Log from that CloudStack station really huge, so I download log from
> > >> > another CloudStack(version 4.10.0) with the same errors.
> > logid:f0c4abda
> > >> for
> > >> > example.
> > >> >
> > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > >> > FirstFitRoutingAllocator) (logid:f0c4abda) Free RAM: 88188272640 ,
> > >> > Requested RAM: 402653184
> > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > >> > FirstFitRoutingAllocator) (logid:f0c4abda) Host has enough CPU and
> RAM
> > >> > available
> > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > >> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc CPU from
> > >> host:
> > >> > 1, used: 48556, reserved: 0, actual total: 144000, total with
> > >> > overprovisioning: 144000; requested
> > cpu:384,alloc_from_last_host?:false
> > >> > ,considerReservedCapacity?: true
> > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > >> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc MEM from
> > >> host:
> > >> > 1, used: 45902462976, reserved: 0, total: 134090735616; requested
> mem:
> > >> > 402653184,alloc_from_last_host?:false ,considerReservedCapacity?:
> true
> > >> > ...
> > >> > 2019-07-25 15:58:07,660 INFO [c.c.v.VirtualMachineManagerImpl]
> > >> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659
> > ctx-fe95df77)
> > >> > (logid:f0c4abda) Unable to start VM on Host[-1-Routing] due to
> failed
> > to
> > >> > get free memory
> > >> > ...
> > >> > 2019-07-25 15:58:14,421 ERROR [c.c.v.VmWorkJobHandlerProxy]
> > >> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659
> > ctx-fe95df77)
> > >> > (logid:f0c4abda) Invocation exception, caused by:
> > >> > com.cloud.exception.InsufficientServerCapacityException: Unable to
> > >> create a
> > >> > deployment for VM[User|i-2-977270-VM]Scope=interface
> > >> > com.cloud.dc.DataCenter; id=3
> > >> >
> > >> > management-server.log https://yadi.sk/d/wzyZVeTKS2kkDQ
> > >> >
> > >> > чт, 25 июл. 2019 г. в 16:51, Andrija Panic <andrija.panic@gmail.com
> >:
> > >> >
> > >> > > Can you share the whole log file (pastebin.org please, or
> similar)
> > ?
> > >> > >
> > >> > > On Wed, 24 Jul 2019 at 17:34, Dmitry Berezhnoy <
> > d.berezhnoy@emzior.ru
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Hello,
> > >> > > >
> > >> > > > Asynchronous creating VM bring to "Unable to start VM on
> > >> > Host[-1-Routing]
> > >> > > > due to failed to get free memory". After that I see ERROR:
> > >> > > > InsufficientServerCapacityException.
> > >> > > > In previous messages I see large amount of free resources:
> > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > >> > ctx-52b48ca6)
> > >> > > > (logid:b6e33d80) Current Used CPU: 115708 , Free CPU:76356
> > >> ,Requested
> > >> > > CPU:
> > >> > > > 512
> > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > >> > ctx-52b48ca6)
> > >> > > > (logid:b6e33d80) Current Used RAM: 122003914752 , Free
> > >> RAM:145667670016
> > >> > > > ,Requested RAM: 536870912
> > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > >> > ctx-52b48ca6)
> > >> > > > (logid:b6e33d80) CPU STATS after allocation: for host: 1, old
> > used:
> > >> > > 115708,
> > >> > > > old reserved: 1536, actual total: 193600, total with
> > >> overprovisioning:
> > >> > > > 193600; new used:116220, reserved:1536; requested
> > >> > > > cpu:512,alloc_from_last:false
> > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > >> > ctx-52b48ca6)
> > >> > > > (logid:b6e33d80) RAM STATS after allocation: for host: 1, old
> > used:
> > >> > > > 122003914752, old reserved: 1610612736, total: 269282197504; new
> > >> used:
> > >> > > > 122540785664, reserved: 1610612736; requested mem:
> > >> > > > 536870912,alloc_from_last:false
> > >> > > >
> > >> > > > How it possible?
> > >> > > >
> > >> > > > Thanks in advance for the help,
> > >> > > > Dmitry.
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > >
> > >> > > Andrija Panić
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >>
> > >> Andrija Panić
> > >>
> > >
> >
>
>
> --
>
> Andrija Panić
>
Re: Failed to get free memory without no reason
Posted by Andrija Panic <an...@gmail.com>.
Based on this (I'm no developer...)
https://github.com/apache/cloudstack/blob/4.11/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L69
...it seems like CloudStack could not fetch free mem from libvirt - i.e. I
would check libvirt logs (you can also try to spin a VM manually via virsh
or Virtual Machine Managed) - or simply restart libvirt daemon.
Could you make a conclusion if this error always happens on some particular
host(s) - libvirt is known to get stuck on very rare occasions - so a
restart would help.
BTW - you seems to be reaching 1.000.000 VMs created in total ??? If so -
that's a nice number :)
Cheers
Andrija
On Thu, 25 Jul 2019 at 17:10, Dmitry Berezhnoy <d....@emzior.ru>
wrote:
> *4.11.2.0
>
> чт, 25 июл. 2019 г. в 18:09, Dmitry Berezhnoy <d....@emzior.ru>:
>
> > First log messages from 10.11.2.0
> >
> > чт, 25 июл. 2019 г. в 17:56, Andrija Panic <an...@gmail.com>:
> >
> >> FYI,4.10 is so broken that it's not considered an official releases
> (check
> >> yourself on the GitHub) - please upgrade to 4.11.3 if possible.
> >>
> >> Will check logs later today.
> >>
> >> Andrija
> >>
> >> On Thu, 25 Jul 2019 at 16:48, Dmitry Berezhnoy <d....@emzior.ru>
> >> wrote:
> >>
> >> > Log from that CloudStack station really huge, so I download log from
> >> > another CloudStack(version 4.10.0) with the same errors.
> logid:f0c4abda
> >> for
> >> > example.
> >> >
> >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> >> > FirstFitRoutingAllocator) (logid:f0c4abda) Free RAM: 88188272640 ,
> >> > Requested RAM: 402653184
> >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> >> > FirstFitRoutingAllocator) (logid:f0c4abda) Host has enough CPU and RAM
> >> > available
> >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> >> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc CPU from
> >> host:
> >> > 1, used: 48556, reserved: 0, actual total: 144000, total with
> >> > overprovisioning: 144000; requested
> cpu:384,alloc_from_last_host?:false
> >> > ,considerReservedCapacity?: true
> >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> >> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc MEM from
> >> host:
> >> > 1, used: 45902462976, reserved: 0, total: 134090735616; requested mem:
> >> > 402653184,alloc_from_last_host?:false ,considerReservedCapacity?: true
> >> > ...
> >> > 2019-07-25 15:58:07,660 INFO [c.c.v.VirtualMachineManagerImpl]
> >> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659
> ctx-fe95df77)
> >> > (logid:f0c4abda) Unable to start VM on Host[-1-Routing] due to failed
> to
> >> > get free memory
> >> > ...
> >> > 2019-07-25 15:58:14,421 ERROR [c.c.v.VmWorkJobHandlerProxy]
> >> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659
> ctx-fe95df77)
> >> > (logid:f0c4abda) Invocation exception, caused by:
> >> > com.cloud.exception.InsufficientServerCapacityException: Unable to
> >> create a
> >> > deployment for VM[User|i-2-977270-VM]Scope=interface
> >> > com.cloud.dc.DataCenter; id=3
> >> >
> >> > management-server.log https://yadi.sk/d/wzyZVeTKS2kkDQ
> >> >
> >> > чт, 25 июл. 2019 г. в 16:51, Andrija Panic <an...@gmail.com>:
> >> >
> >> > > Can you share the whole log file (pastebin.org please, or similar)
> ?
> >> > >
> >> > > On Wed, 24 Jul 2019 at 17:34, Dmitry Berezhnoy <
> d.berezhnoy@emzior.ru
> >> >
> >> > > wrote:
> >> > >
> >> > > > Hello,
> >> > > >
> >> > > > Asynchronous creating VM bring to "Unable to start VM on
> >> > Host[-1-Routing]
> >> > > > due to failed to get free memory". After that I see ERROR:
> >> > > > InsufficientServerCapacityException.
> >> > > > In previous messages I see large amount of free resources:
> >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> >> > ctx-52b48ca6)
> >> > > > (logid:b6e33d80) Current Used CPU: 115708 , Free CPU:76356
> >> ,Requested
> >> > > CPU:
> >> > > > 512
> >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> >> > ctx-52b48ca6)
> >> > > > (logid:b6e33d80) Current Used RAM: 122003914752 , Free
> >> RAM:145667670016
> >> > > > ,Requested RAM: 536870912
> >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> >> > ctx-52b48ca6)
> >> > > > (logid:b6e33d80) CPU STATS after allocation: for host: 1, old
> used:
> >> > > 115708,
> >> > > > old reserved: 1536, actual total: 193600, total with
> >> overprovisioning:
> >> > > > 193600; new used:116220, reserved:1536; requested
> >> > > > cpu:512,alloc_from_last:false
> >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> >> > ctx-52b48ca6)
> >> > > > (logid:b6e33d80) RAM STATS after allocation: for host: 1, old
> used:
> >> > > > 122003914752, old reserved: 1610612736, total: 269282197504; new
> >> used:
> >> > > > 122540785664, reserved: 1610612736; requested mem:
> >> > > > 536870912,alloc_from_last:false
> >> > > >
> >> > > > How it possible?
> >> > > >
> >> > > > Thanks in advance for the help,
> >> > > > Dmitry.
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Andrija Panić
> >> > >
> >> >
> >>
> >>
> >> --
> >>
> >> Andrija Panić
> >>
> >
>
--
Andrija Panić
Re: Failed to get free memory without no reason
Posted by Dmitry Berezhnoy <d....@emzior.ru>.
*4.11.2.0
чт, 25 июл. 2019 г. в 18:09, Dmitry Berezhnoy <d....@emzior.ru>:
> First log messages from 10.11.2.0
>
> чт, 25 июл. 2019 г. в 17:56, Andrija Panic <an...@gmail.com>:
>
>> FYI,4.10 is so broken that it's not considered an official releases (check
>> yourself on the GitHub) - please upgrade to 4.11.3 if possible.
>>
>> Will check logs later today.
>>
>> Andrija
>>
>> On Thu, 25 Jul 2019 at 16:48, Dmitry Berezhnoy <d....@emzior.ru>
>> wrote:
>>
>> > Log from that CloudStack station really huge, so I download log from
>> > another CloudStack(version 4.10.0) with the same errors. logid:f0c4abda
>> for
>> > example.
>> >
>> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
>> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
>> > FirstFitRoutingAllocator) (logid:f0c4abda) Free RAM: 88188272640 ,
>> > Requested RAM: 402653184
>> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
>> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
>> > FirstFitRoutingAllocator) (logid:f0c4abda) Host has enough CPU and RAM
>> > available
>> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
>> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
>> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc CPU from
>> host:
>> > 1, used: 48556, reserved: 0, actual total: 144000, total with
>> > overprovisioning: 144000; requested cpu:384,alloc_from_last_host?:false
>> > ,considerReservedCapacity?: true
>> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
>> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
>> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc MEM from
>> host:
>> > 1, used: 45902462976, reserved: 0, total: 134090735616; requested mem:
>> > 402653184,alloc_from_last_host?:false ,considerReservedCapacity?: true
>> > ...
>> > 2019-07-25 15:58:07,660 INFO [c.c.v.VirtualMachineManagerImpl]
>> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659 ctx-fe95df77)
>> > (logid:f0c4abda) Unable to start VM on Host[-1-Routing] due to failed to
>> > get free memory
>> > ...
>> > 2019-07-25 15:58:14,421 ERROR [c.c.v.VmWorkJobHandlerProxy]
>> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659 ctx-fe95df77)
>> > (logid:f0c4abda) Invocation exception, caused by:
>> > com.cloud.exception.InsufficientServerCapacityException: Unable to
>> create a
>> > deployment for VM[User|i-2-977270-VM]Scope=interface
>> > com.cloud.dc.DataCenter; id=3
>> >
>> > management-server.log https://yadi.sk/d/wzyZVeTKS2kkDQ
>> >
>> > чт, 25 июл. 2019 г. в 16:51, Andrija Panic <an...@gmail.com>:
>> >
>> > > Can you share the whole log file (pastebin.org please, or similar) ?
>> > >
>> > > On Wed, 24 Jul 2019 at 17:34, Dmitry Berezhnoy <d.berezhnoy@emzior.ru
>> >
>> > > wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > Asynchronous creating VM bring to "Unable to start VM on
>> > Host[-1-Routing]
>> > > > due to failed to get free memory". After that I see ERROR:
>> > > > InsufficientServerCapacityException.
>> > > > In previous messages I see large amount of free resources:
>> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
>> > ctx-52b48ca6)
>> > > > (logid:b6e33d80) Current Used CPU: 115708 , Free CPU:76356
>> ,Requested
>> > > CPU:
>> > > > 512
>> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
>> > ctx-52b48ca6)
>> > > > (logid:b6e33d80) Current Used RAM: 122003914752 , Free
>> RAM:145667670016
>> > > > ,Requested RAM: 536870912
>> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
>> > ctx-52b48ca6)
>> > > > (logid:b6e33d80) CPU STATS after allocation: for host: 1, old used:
>> > > 115708,
>> > > > old reserved: 1536, actual total: 193600, total with
>> overprovisioning:
>> > > > 193600; new used:116220, reserved:1536; requested
>> > > > cpu:512,alloc_from_last:false
>> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
>> > ctx-52b48ca6)
>> > > > (logid:b6e33d80) RAM STATS after allocation: for host: 1, old used:
>> > > > 122003914752, old reserved: 1610612736, total: 269282197504; new
>> used:
>> > > > 122540785664, reserved: 1610612736; requested mem:
>> > > > 536870912,alloc_from_last:false
>> > > >
>> > > > How it possible?
>> > > >
>> > > > Thanks in advance for the help,
>> > > > Dmitry.
>> > > >
>> > >
>> > >
>> > > --
>> > >
>> > > Andrija Panić
>> > >
>> >
>>
>>
>> --
>>
>> Andrija Panić
>>
>
Re: Failed to get free memory without no reason
Posted by Dmitry Berezhnoy <d....@emzior.ru>.
First log messages from 10.11.2.0
чт, 25 июл. 2019 г. в 17:56, Andrija Panic <an...@gmail.com>:
> FYI,4.10 is so broken that it's not considered an official releases (check
> yourself on the GitHub) - please upgrade to 4.11.3 if possible.
>
> Will check logs later today.
>
> Andrija
>
> On Thu, 25 Jul 2019 at 16:48, Dmitry Berezhnoy <d....@emzior.ru>
> wrote:
>
> > Log from that CloudStack station really huge, so I download log from
> > another CloudStack(version 4.10.0) with the same errors. logid:f0c4abda
> for
> > example.
> >
> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > FirstFitRoutingAllocator) (logid:f0c4abda) Free RAM: 88188272640 ,
> > Requested RAM: 402653184
> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > FirstFitRoutingAllocator) (logid:f0c4abda) Host has enough CPU and RAM
> > available
> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc CPU from
> host:
> > 1, used: 48556, reserved: 0, actual total: 144000, total with
> > overprovisioning: 144000; requested cpu:384,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc MEM from
> host:
> > 1, used: 45902462976, reserved: 0, total: 134090735616; requested mem:
> > 402653184,alloc_from_last_host?:false ,considerReservedCapacity?: true
> > ...
> > 2019-07-25 15:58:07,660 INFO [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659 ctx-fe95df77)
> > (logid:f0c4abda) Unable to start VM on Host[-1-Routing] due to failed to
> > get free memory
> > ...
> > 2019-07-25 15:58:14,421 ERROR [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659 ctx-fe95df77)
> > (logid:f0c4abda) Invocation exception, caused by:
> > com.cloud.exception.InsufficientServerCapacityException: Unable to
> create a
> > deployment for VM[User|i-2-977270-VM]Scope=interface
> > com.cloud.dc.DataCenter; id=3
> >
> > management-server.log https://yadi.sk/d/wzyZVeTKS2kkDQ
> >
> > чт, 25 июл. 2019 г. в 16:51, Andrija Panic <an...@gmail.com>:
> >
> > > Can you share the whole log file (pastebin.org please, or similar) ?
> > >
> > > On Wed, 24 Jul 2019 at 17:34, Dmitry Berezhnoy <d....@emzior.ru>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > Asynchronous creating VM bring to "Unable to start VM on
> > Host[-1-Routing]
> > > > due to failed to get free memory". After that I see ERROR:
> > > > InsufficientServerCapacityException.
> > > > In previous messages I see large amount of free resources:
> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > ctx-52b48ca6)
> > > > (logid:b6e33d80) Current Used CPU: 115708 , Free CPU:76356 ,Requested
> > > CPU:
> > > > 512
> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > ctx-52b48ca6)
> > > > (logid:b6e33d80) Current Used RAM: 122003914752 , Free
> RAM:145667670016
> > > > ,Requested RAM: 536870912
> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > ctx-52b48ca6)
> > > > (logid:b6e33d80) CPU STATS after allocation: for host: 1, old used:
> > > 115708,
> > > > old reserved: 1536, actual total: 193600, total with
> overprovisioning:
> > > > 193600; new used:116220, reserved:1536; requested
> > > > cpu:512,alloc_from_last:false
> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> > ctx-52b48ca6)
> > > > (logid:b6e33d80) RAM STATS after allocation: for host: 1, old used:
> > > > 122003914752, old reserved: 1610612736, total: 269282197504; new
> used:
> > > > 122540785664, reserved: 1610612736; requested mem:
> > > > 536870912,alloc_from_last:false
> > > >
> > > > How it possible?
> > > >
> > > > Thanks in advance for the help,
> > > > Dmitry.
> > > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
>
>
> --
>
> Andrija Panić
>
Re: Failed to get free memory without no reason
Posted by Andrija Panic <an...@gmail.com>.
FYI,4.10 is so broken that it's not considered an official releases (check
yourself on the GitHub) - please upgrade to 4.11.3 if possible.
Will check logs later today.
Andrija
On Thu, 25 Jul 2019 at 16:48, Dmitry Berezhnoy <d....@emzior.ru>
wrote:
> Log from that CloudStack station really huge, so I download log from
> another CloudStack(version 4.10.0) with the same errors. logid:f0c4abda for
> example.
>
> 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> FirstFitRoutingAllocator) (logid:f0c4abda) Free RAM: 88188272640 ,
> Requested RAM: 402653184
> 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> FirstFitRoutingAllocator) (logid:f0c4abda) Host has enough CPU and RAM
> available
> 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc CPU from host:
> 1, used: 48556, reserved: 0, actual total: 144000, total with
> overprovisioning: 144000; requested cpu:384,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
> FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc MEM from host:
> 1, used: 45902462976, reserved: 0, total: 134090735616; requested mem:
> 402653184,alloc_from_last_host?:false ,considerReservedCapacity?: true
> ...
> 2019-07-25 15:58:07,660 INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659 ctx-fe95df77)
> (logid:f0c4abda) Unable to start VM on Host[-1-Routing] due to failed to
> get free memory
> ...
> 2019-07-25 15:58:14,421 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659 ctx-fe95df77)
> (logid:f0c4abda) Invocation exception, caused by:
> com.cloud.exception.InsufficientServerCapacityException: Unable to create a
> deployment for VM[User|i-2-977270-VM]Scope=interface
> com.cloud.dc.DataCenter; id=3
>
> management-server.log https://yadi.sk/d/wzyZVeTKS2kkDQ
>
> чт, 25 июл. 2019 г. в 16:51, Andrija Panic <an...@gmail.com>:
>
> > Can you share the whole log file (pastebin.org please, or similar) ?
> >
> > On Wed, 24 Jul 2019 at 17:34, Dmitry Berezhnoy <d....@emzior.ru>
> > wrote:
> >
> > > Hello,
> > >
> > > Asynchronous creating VM bring to "Unable to start VM on
> Host[-1-Routing]
> > > due to failed to get free memory". After that I see ERROR:
> > > InsufficientServerCapacityException.
> > > In previous messages I see large amount of free resources:
> > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> ctx-52b48ca6)
> > > (logid:b6e33d80) Current Used CPU: 115708 , Free CPU:76356 ,Requested
> > CPU:
> > > 512
> > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> ctx-52b48ca6)
> > > (logid:b6e33d80) Current Used RAM: 122003914752 , Free RAM:145667670016
> > > ,Requested RAM: 536870912
> > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> ctx-52b48ca6)
> > > (logid:b6e33d80) CPU STATS after allocation: for host: 1, old used:
> > 115708,
> > > old reserved: 1536, actual total: 193600, total with overprovisioning:
> > > 193600; new used:116220, reserved:1536; requested
> > > cpu:512,alloc_from_last:false
> > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455
> ctx-52b48ca6)
> > > (logid:b6e33d80) RAM STATS after allocation: for host: 1, old used:
> > > 122003914752, old reserved: 1610612736, total: 269282197504; new used:
> > > 122540785664, reserved: 1610612736; requested mem:
> > > 536870912,alloc_from_last:false
> > >
> > > How it possible?
> > >
> > > Thanks in advance for the help,
> > > Dmitry.
> > >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
--
Andrija Panić
Re: Failed to get free memory without no reason
Posted by Dmitry Berezhnoy <d....@emzior.ru>.
Log from that CloudStack station really huge, so I download log from
another CloudStack(version 4.10.0) with the same errors. logid:f0c4abda for
example.
2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
FirstFitRoutingAllocator) (logid:f0c4abda) Free RAM: 88188272640 ,
Requested RAM: 402653184
2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
FirstFitRoutingAllocator) (logid:f0c4abda) Host has enough CPU and RAM
available
2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc CPU from host:
1, used: 48556, reserved: 0, actual total: 144000, total with
overprovisioning: 144000; requested cpu:384,alloc_from_last_host?:false
,considerReservedCapacity?: true
2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7
FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc MEM from host:
1, used: 45902462976, reserved: 0, total: 134090735616; requested mem:
402653184,alloc_from_last_host?:false ,considerReservedCapacity?: true
...
2019-07-25 15:58:07,660 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659 ctx-fe95df77)
(logid:f0c4abda) Unable to start VM on Host[-1-Routing] due to failed to
get free memory
...
2019-07-25 15:58:14,421 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659 ctx-fe95df77)
(logid:f0c4abda) Invocation exception, caused by:
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[User|i-2-977270-VM]Scope=interface
com.cloud.dc.DataCenter; id=3
management-server.log https://yadi.sk/d/wzyZVeTKS2kkDQ
чт, 25 июл. 2019 г. в 16:51, Andrija Panic <an...@gmail.com>:
> Can you share the whole log file (pastebin.org please, or similar) ?
>
> On Wed, 24 Jul 2019 at 17:34, Dmitry Berezhnoy <d....@emzior.ru>
> wrote:
>
> > Hello,
> >
> > Asynchronous creating VM bring to "Unable to start VM on Host[-1-Routing]
> > due to failed to get free memory". After that I see ERROR:
> > InsufficientServerCapacityException.
> > In previous messages I see large amount of free resources:
> > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
> > (logid:b6e33d80) Current Used CPU: 115708 , Free CPU:76356 ,Requested
> CPU:
> > 512
> > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
> > (logid:b6e33d80) Current Used RAM: 122003914752 , Free RAM:145667670016
> > ,Requested RAM: 536870912
> > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
> > (logid:b6e33d80) CPU STATS after allocation: for host: 1, old used:
> 115708,
> > old reserved: 1536, actual total: 193600, total with overprovisioning:
> > 193600; new used:116220, reserved:1536; requested
> > cpu:512,alloc_from_last:false
> > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
> > (logid:b6e33d80) RAM STATS after allocation: for host: 1, old used:
> > 122003914752, old reserved: 1610612736, total: 269282197504; new used:
> > 122540785664, reserved: 1610612736; requested mem:
> > 536870912,alloc_from_last:false
> >
> > How it possible?
> >
> > Thanks in advance for the help,
> > Dmitry.
> >
>
>
> --
>
> Andrija Panić
>
Re: Failed to get free memory without no reason
Posted by Andrija Panic <an...@gmail.com>.
Can you share the whole log file (pastebin.org please, or similar) ?
On Wed, 24 Jul 2019 at 17:34, Dmitry Berezhnoy <d....@emzior.ru>
wrote:
> Hello,
>
> Asynchronous creating VM bring to "Unable to start VM on Host[-1-Routing]
> due to failed to get free memory". After that I see ERROR:
> InsufficientServerCapacityException.
> In previous messages I see large amount of free resources:
> 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
> (logid:b6e33d80) Current Used CPU: 115708 , Free CPU:76356 ,Requested CPU:
> 512
> 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
> (logid:b6e33d80) Current Used RAM: 122003914752 , Free RAM:145667670016
> ,Requested RAM: 536870912
> 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
> (logid:b6e33d80) CPU STATS after allocation: for host: 1, old used: 115708,
> old reserved: 1536, actual total: 193600, total with overprovisioning:
> 193600; new used:116220, reserved:1536; requested
> cpu:512,alloc_from_last:false
> 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 ctx-52b48ca6)
> (logid:b6e33d80) RAM STATS after allocation: for host: 1, old used:
> 122003914752, old reserved: 1610612736, total: 269282197504; new used:
> 122540785664, reserved: 1610612736; requested mem:
> 536870912,alloc_from_last:false
>
> How it possible?
>
> Thanks in advance for the help,
> Dmitry.
>
--
Andrija Panić