You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Soheil Eizadi <se...@infoblox.com> on 2013/10/16 20:56:58 UTC
XenServer Memory Increase Problem
I am running on CloudStack Master 4.3.
I had a use case where my XenServer ran out of memory and I could not create any more instances on CloudStack.
I shutdown the XenServer and increased the memory on XenServer and brought up the Management Server again, but the Management Server still sees the old values. How is this suppose to work?
As a workaround I assume I could find the database entry for the old RAM value and update with the new Memory Total.
Here is some more detailed logs:
The stats for the XenServer from CloudStack:
Total CPU 2 x 2.67 GHz
CPU Utilized 0.08%
CPU Allocated for VMs 0%
Memory Total 7.03 GB
Memory Allocated 2.50 GB
Memory Used 3.60 MB
Network Read 558.23 MB
Network Write 23.20 MB
The log of memory problem:
WARN [o.a.c.alerts] (Job-Executor-5:ctx-3b5b1c95) alertType:: 8 // dataCenterId:: 1 // podId:: null // clusterId:: null // message:: Failed to deploy Vm with Id: 27, on Host wi
th Id: null
INFO [o.a.c.a.c.u.v.DeployVMCmd] (Job-Executor-5:ctx-3b5b1c95) com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|test-2]Scope=in
terface com.cloud.dc.DataCenter; id=1
INFO [o.a.c.a.c.u.v.DeployVMCmd] (Job-Executor-5:ctx-3b5b1c95) Unable to create a deployment for VM[User|test-2]
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|test-2]Scope=interface com.cloud.dc.DataCenter; id=1
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:210)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:198)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3372)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2940)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:421)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:97)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:526)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-5:null) Remove job-62 from job monitoring
RE: XenServer Memory Increase Problem
Posted by Soheil Eizadi <se...@infoblox.com>.
Here is more detail for others that have this bug, until it is fixed:
Find the total memory listed for XenServer it is under column "ram" for the XenServer you have:
mysql> select * from host;
Then find the XenServer above and in another table op_host_capacity update column total_capacity
mysql> select * from op_host_capacity;
mysql> update op_host_capacity set total_capacity=7546932288 where id=1;
mysql> commit;
________________________________________
From: Soheil Eizadi [seizadi@infoblox.com]
Sent: Wednesday, October 16, 2013 12:25 PM
To: dev@cloudstack.apache.org
Subject: RE: XenServer Memory Increase Problem
Thanks Nitin, I created Jira Bug:
https://issues.apache.org/jira/browse/CLOUDSTACK-4883
-Soheil
________________________________________
From: Nitin Mehta [Nitin.Mehta@citrix.com]
Sent: Wednesday, October 16, 2013 12:15 PM
To: dev@cloudstack.apache.org
Subject: Re: XenServer Memory Increase Problem
Soheil - I guess this is a bug that addition of new memory isn't detected.
File it if not already done.
I would suggest you to change the DB as of now. Right tables are host and
op_host_capacity (capacity_type=0).
You might have to keep in mind the over provisioning factors you have
kept. See if changing it works.
On 16/10/13 11:56 AM, "Soheil Eizadi" <se...@infoblox.com> wrote:
>I am running on CloudStack Master 4.3.
>
>I had a use case where my XenServer ran out of memory and I could not
>create any more instances on CloudStack.
>I shutdown the XenServer and increased the memory on XenServer and
>brought up the Management Server again, but the Management Server still
>sees the old values. How is this suppose to work?
>
>As a workaround I assume I could find the database entry for the old RAM
>value and update with the new Memory Total.
>
>Here is some more detailed logs:
>
>The stats for the XenServer from CloudStack:
>
>Total CPU 2 x 2.67 GHz
>CPU Utilized 0.08%
>CPU Allocated for VMs 0%
>Memory Total 7.03 GB
>Memory Allocated 2.50 GB
>Memory Used 3.60 MB
>Network Read 558.23 MB
>Network Write 23.20 MB
>
>
>The log of memory problem:
>WARN [o.a.c.alerts] (Job-Executor-5:ctx-3b5b1c95) alertType:: 8 //
>dataCenterId:: 1 // podId:: null // clusterId:: null // message:: Failed
>to deploy Vm with Id: 27, on Host wi
>th Id: null
>INFO [o.a.c.a.c.u.v.DeployVMCmd] (Job-Executor-5:ctx-3b5b1c95)
>com.cloud.exception.InsufficientServerCapacityException: Unable to create
>a deployment for VM[User|test-2]Scope=in
>terface com.cloud.dc.DataCenter; id=1
>INFO [o.a.c.a.c.u.v.DeployVMCmd] (Job-Executor-5:ctx-3b5b1c95) Unable to
>create a deployment for VM[User|test-2]
>com.cloud.exception.InsufficientServerCapacityException: Unable to create
>a deployment for VM[User|test-2]Scope=interface com.cloud.dc.DataCenter;
>id=1
> at
>org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveV
>irtualMachine(VMEntityManagerImpl.java:210)
> at
>org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.res
>erve(VirtualMachineEntityImpl.java:198)
> at
>com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:
>3372)
> at
>com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:
>2954)
> at
>com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:
>2940)
> at
>com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.
>java:421)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
> at
>com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:97)
> at
>org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.run(AsyncJ
>obManagerImpl.java:526)
> at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>1110)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:603)
> at java.lang.Thread.run(Thread.java:679)
>INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-5:null) Remove job-62
>from job monitoring
>
RE: XenServer Memory Increase Problem
Posted by Soheil Eizadi <se...@infoblox.com>.
Thanks Nitin, I created Jira Bug:
https://issues.apache.org/jira/browse/CLOUDSTACK-4883
-Soheil
________________________________________
From: Nitin Mehta [Nitin.Mehta@citrix.com]
Sent: Wednesday, October 16, 2013 12:15 PM
To: dev@cloudstack.apache.org
Subject: Re: XenServer Memory Increase Problem
Soheil - I guess this is a bug that addition of new memory isn't detected.
File it if not already done.
I would suggest you to change the DB as of now. Right tables are host and
op_host_capacity (capacity_type=0).
You might have to keep in mind the over provisioning factors you have
kept. See if changing it works.
On 16/10/13 11:56 AM, "Soheil Eizadi" <se...@infoblox.com> wrote:
>I am running on CloudStack Master 4.3.
>
>I had a use case where my XenServer ran out of memory and I could not
>create any more instances on CloudStack.
>I shutdown the XenServer and increased the memory on XenServer and
>brought up the Management Server again, but the Management Server still
>sees the old values. How is this suppose to work?
>
>As a workaround I assume I could find the database entry for the old RAM
>value and update with the new Memory Total.
>
>Here is some more detailed logs:
>
>The stats for the XenServer from CloudStack:
>
>Total CPU 2 x 2.67 GHz
>CPU Utilized 0.08%
>CPU Allocated for VMs 0%
>Memory Total 7.03 GB
>Memory Allocated 2.50 GB
>Memory Used 3.60 MB
>Network Read 558.23 MB
>Network Write 23.20 MB
>
>
>The log of memory problem:
>WARN [o.a.c.alerts] (Job-Executor-5:ctx-3b5b1c95) alertType:: 8 //
>dataCenterId:: 1 // podId:: null // clusterId:: null // message:: Failed
>to deploy Vm with Id: 27, on Host wi
>th Id: null
>INFO [o.a.c.a.c.u.v.DeployVMCmd] (Job-Executor-5:ctx-3b5b1c95)
>com.cloud.exception.InsufficientServerCapacityException: Unable to create
>a deployment for VM[User|test-2]Scope=in
>terface com.cloud.dc.DataCenter; id=1
>INFO [o.a.c.a.c.u.v.DeployVMCmd] (Job-Executor-5:ctx-3b5b1c95) Unable to
>create a deployment for VM[User|test-2]
>com.cloud.exception.InsufficientServerCapacityException: Unable to create
>a deployment for VM[User|test-2]Scope=interface com.cloud.dc.DataCenter;
>id=1
> at
>org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveV
>irtualMachine(VMEntityManagerImpl.java:210)
> at
>org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.res
>erve(VirtualMachineEntityImpl.java:198)
> at
>com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:
>3372)
> at
>com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:
>2954)
> at
>com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:
>2940)
> at
>com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.
>java:421)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
> at
>com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:97)
> at
>org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.run(AsyncJ
>obManagerImpl.java:526)
> at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>1110)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:603)
> at java.lang.Thread.run(Thread.java:679)
>INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-5:null) Remove job-62
>from job monitoring
>
Re: XenServer Memory Increase Problem
Posted by Nitin Mehta <Ni...@citrix.com>.
Soheil - I guess this is a bug that addition of new memory isn't detected.
File it if not already done.
I would suggest you to change the DB as of now. Right tables are host and
op_host_capacity (capacity_type=0).
You might have to keep in mind the over provisioning factors you have
kept. See if changing it works.
On 16/10/13 11:56 AM, "Soheil Eizadi" <se...@infoblox.com> wrote:
>I am running on CloudStack Master 4.3.
>
>I had a use case where my XenServer ran out of memory and I could not
>create any more instances on CloudStack.
>I shutdown the XenServer and increased the memory on XenServer and
>brought up the Management Server again, but the Management Server still
>sees the old values. How is this suppose to work?
>
>As a workaround I assume I could find the database entry for the old RAM
>value and update with the new Memory Total.
>
>Here is some more detailed logs:
>
>The stats for the XenServer from CloudStack:
>
>Total CPU 2 x 2.67 GHz
>CPU Utilized 0.08%
>CPU Allocated for VMs 0%
>Memory Total 7.03 GB
>Memory Allocated 2.50 GB
>Memory Used 3.60 MB
>Network Read 558.23 MB
>Network Write 23.20 MB
>
>
>The log of memory problem:
>WARN [o.a.c.alerts] (Job-Executor-5:ctx-3b5b1c95) alertType:: 8 //
>dataCenterId:: 1 // podId:: null // clusterId:: null // message:: Failed
>to deploy Vm with Id: 27, on Host wi
>th Id: null
>INFO [o.a.c.a.c.u.v.DeployVMCmd] (Job-Executor-5:ctx-3b5b1c95)
>com.cloud.exception.InsufficientServerCapacityException: Unable to create
>a deployment for VM[User|test-2]Scope=in
>terface com.cloud.dc.DataCenter; id=1
>INFO [o.a.c.a.c.u.v.DeployVMCmd] (Job-Executor-5:ctx-3b5b1c95) Unable to
>create a deployment for VM[User|test-2]
>com.cloud.exception.InsufficientServerCapacityException: Unable to create
>a deployment for VM[User|test-2]Scope=interface com.cloud.dc.DataCenter;
>id=1
> at
>org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveV
>irtualMachine(VMEntityManagerImpl.java:210)
> at
>org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.res
>erve(VirtualMachineEntityImpl.java:198)
> at
>com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:
>3372)
> at
>com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:
>2954)
> at
>com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:
>2940)
> at
>com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD
>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.
>java:421)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
> at
>com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:97)
> at
>org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.run(AsyncJ
>obManagerImpl.java:526)
> at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>1110)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:603)
> at java.lang.Thread.run(Thread.java:679)
>INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-5:null) Remove job-62
>from job monitoring
>