You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Ugo Vasi <ug...@procne.it.INVALID> on 2018/12/13 08:01:33 UTC
URGENT: Unable to schedule async job for command
com.cloud.vm.VmWorkStart, unexpected exception
Hi all,
I'm trying to reboot a vm after the host it ran on crashed and restarted
from the HA system. All the VMs running on the rebooted host were
restarted on other hosts except one.
In the web interface and using cloudmonkey I get this message:
"Unable to schedule async job for command com.cloud.vm.VmWorkStart,
unexpected exception."
In the management-server.log file there would seem to be a problem when
creating an element that is duplicated (Duplicate entry 'xxxx' for key
'PRIMARY'):
2018-12-13 08:44:06,659 DEBUG [c.c.a.ApiServlet]
(qtp1096283470-445:ctx-6c065e06) (logid:87edf8d7) ===START=== 10.80.0.6
-- GET
command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
2018-12-13 08:44:06,665 DEBUG [c.c.a.ApiServer]
(qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) CIDRs
from which account 'Acct[26e597f2-b5ca-11e8-a619-c8cbb8cb15cd-admin]' is
allowed to perform API calls: 0.0.0.0/0,::/0
2018-12-13 08:44:06,693 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:7e0c4dc9) Add job-1343
into job monitoring
2018-12-13 08:44:06,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) submit
async job-1343, details: AsyncJobVO {id:1343, userId: 2, accountId: 2,
instanceType: VirtualMachine, instanceId: 8, cmd:
org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
{"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 220777304233416, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null}
2018-12-13 08:44:06,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Executing
AsyncJobVO {id:1343, userId: 2, accountId: 2, instanceType:
VirtualMachine, instanceId: 8, cmd:
org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
{"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 220777304233416, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null}
2018-12-13 08:44:06,705 DEBUG [c.c.a.ApiServlet]
(qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7)
===END=== 10.80.0.6 -- GET
command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
2018-12-13 08:44:06,745 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Service SecurityGroup is not supported in the network id=205
2018-12-13 08:44:06,752 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Service SecurityGroup is not supported in the network id=205
2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
DeploymentPlanner allocation algorithm: null
2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Trying to allocate a host and storage pools from dc:1,
pod:null,cluster:null, requested cpu: 8000, requested ram: 8594128896
2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Is ROOT volume READY (pool already allocated)?: Yes
2018-12-13 08:44:06,783 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Deploy avoids pods: [], clusters: [], hosts: []
2018-12-13 08:44:06,784 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
This VM has last host_id specified, trying to choose the same host: 10
2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Host: 10 has cpu capability (cpu:12, speed:3000) to support requested
CPU: 4 and requested speed: 2000
2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Checking if host: 10 has enough capacity for requested CPU: 8000 and
requested RAM: 8594128896 , cpuOverprovisioningFactor: 1.0
2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Hosts's actual total CPU: 36000 and CPU after applying overprovisioning:
36000
2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
We need to allocate to the last host again, so checking if there is
enough reserved capacity
2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Reserved CPU: 8000 , Requested CPU: 8000
2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Reserved RAM: 8594128896 , Requested RAM: 8594128896
2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Host has enough CPU and RAM available
2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
STATS: Can alloc CPU from host: 10, used: 4500, reserved: 8000, actual
total: 36000, total with overprovisioning: 36000; requested
cpu:8000,alloc_from_last_host?:true ,considerReservedCapacity?: true
2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
STATS: Can alloc MEM from host: 10, used: 2415919104, reserved:
8594128896, total: 49554284544; requested mem:
8594128896,alloc_from_last_host?:true ,considerReservedCapacity?: true
2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
The last host of this VM is UP and has enough capacity
2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Now checking for suitable pools under zone: 1, pod: 1, cluster: 1
2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Checking suitable pools for volume (Id, Type): (8,ROOT)
2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Volume has pool already allocated, checking if pool can be reused, poolId: 1
2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Planner need not allocate a pool for this volume since its READY
2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Checking suitable pools for volume (Id, Type): (12,DATADISK)
2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Volume has pool already allocated, checking if pool can be reused, poolId: 1
2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Planner need not allocate a pool for this volume since its READY
2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Checking suitable pools for volume (Id, Type): (17,DATADISK)
2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Volume has pool already allocated, checking if pool can be reused, poolId: 1
2018-12-13 08:44:06,819 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Planner need not allocate a pool for this volume since its READY
2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Checking suitable pools for volume (Id, Type): (58,DATADISK)
2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Volume has pool already allocated, checking if pool can be reused, poolId: 1
2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Planner need not allocate a pool for this volume since its READY
2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Trying to find a potenial host and associated storage pools from the
suitable host/pool lists for this VM
2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Checking if host: 10 can access any suitable storage pool for volume:
DATADISK
2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Host: 10 can access pool: 1
2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Checking if host: 10 can access any suitable storage pool for volume:
DATADISK
2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Host: 10 can access pool: 1
2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Checking if host: 10 can access any suitable storage pool for volume:
DATADISK
2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Host: 10 can access pool: 1
2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Checking if host: 10 can access any suitable storage pool for volume: ROOT
2018-12-13 08:44:06,831 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Host: 10 can access pool: 1
2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Found a potential host id: 10 name: cshp143 and associated storage pools
for this VM
2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Returning Deployment Destination:
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(1)-Pod(1)-Cluster(1)-Host(10)-Storage()]
2018-12-13 08:44:06,856 DEBUG [c.c.u.d.T.Transaction]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Rolling back the transaction: Time = 3 Name = API-Job-Executor-8; called
by
-TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:233-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
2018-12-13 08:44:06,861 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Unable to schedule async job for command com.cloud.vm.VmWorkStart,
unexpected exception.
javax.persistence.EntityExistsException: Entity already exists:
at
com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
at
com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
at
com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:233)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
at
org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Duplicate entry '1344' for key 'PRIMARY'
at
sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
... 59 more
2018-12-13 08:44:06,876 DEBUG [c.c.u.d.T.Transaction]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Rolling back the transaction: Time = 2 Name = API-Job-Executor-8; called
by
-TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:245-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
2018-12-13 08:44:06,880 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
Unable to schedule async job for command com.cloud.vm.VmWorkStart,
unexpected exception.
javax.persistence.EntityExistsException: Entity already exists:
at
com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
at
com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
at
com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
at
org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Duplicate entry '1345' for key 'PRIMARY'
at
sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
... 59 more
2018-12-13 08:44:06,889 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Unable to schedule
async job for command com.cloud.vm.VmWorkStart, unexpected exception.
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
at
com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
at
org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-12-13 08:44:06,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Complete
async job-1343, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
to schedule async job for command com.cloud.vm.VmWorkStart, unexpected
exception."}
2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Publish
async job-1343 complete on message bus
2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up jobs
related to job-1343
2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Update db
status for job-1343
2018-12-13 08:44:06,895 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up jobs
joined with job-1343 and disjoin all subjobs created from job- 1343
2018-12-13 08:44:06,899 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Done
executing org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
for job-1343
2018-12-13 08:44:06,899 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Remove
job-1343 from job monitoring
2018-12-13 08:44:07,056 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) Begin cleanup
expired async-jobs
2018-12-13 08:44:07,062 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) End cleanup
expired async-jobs
This problem has already appeared to me other times (in other
circumstances) and I have reported it but I have not found a solution.
Can you help me identify the problem?
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.
Re: URGENT: Unable to schedule async job for command
com.cloud.vm.VmWorkStart, unexpected exception
Posted by Rene Moser <ma...@renemoser.net>.
Hi
On 12/13/18 11:50 AM, Ugo Vasi wrote:
> Hi René ,
> the cloustack installation is 4.11.1.0. From the issue you reported to
> me I do not understand if the problem has been solved or not..
>
> The big problem is that I can not perform any new jobs.
>
> In addition to the stopped VM there are two routes that were running on
> the host that has crashed.
>
> One of these routers was redundant as a master of one of the isolated
> networks along with another that has taken its place and is now working.
As far as I can see, this issue is not solved. It is exactly the same
issue we encountered. It seems, that after a while "job timeout? job
cleanup time?" you will be able to run jobs again.
The guys from shapeblue should know more.
Regards
René
Re: URGENT: Unable to schedule async job for command
com.cloud.vm.VmWorkStart, unexpected exception
Posted by Ugo Vasi <ug...@procne.it.INVALID>.
Hi René ,
the cloustack installation is 4.11.1.0. From the issue you reported to
me I do not understand if the problem has been solved or not..
The big problem is that I can not perform any new jobs.
In addition to the stopped VM there are two routes that were running on
the host that has crashed.
One of these routers was redundant as a master of one of the isolated
networks along with another that has taken its place and is now working.
Il 13/12/18 11:07, Rene Moser ha scritto:
> Hi
>
> I think you hit https://github.com/apache/cloudstack/issues/2880
>
> Is this already 4.11.2?
>
> We have also seen this in our lab, could not really reproduce it, seems
> to be a race condition.
>
> Regards
> René
>
> On 12/13/18 9:01 AM, Ugo Vasi wrote:
>> Hi all,
>> I'm trying to reboot a vm after the host it ran on crashed and restarted
>> from the HA system. All the VMs running on the rebooted host were
>> restarted on other hosts except one.
>> In the web interface and using cloudmonkey I get this message:
>> "Unable to schedule async job for command com.cloud.vm.VmWorkStart,
>> unexpected exception."
>>
>> In the management-server.log file there would seem to be a problem when
>> creating an element that is duplicated (Duplicate entry 'xxxx' for key
>> 'PRIMARY'):
>>
>> 2018-12-13 08:44:06,659 DEBUG [c.c.a.ApiServlet]
>> (qtp1096283470-445:ctx-6c065e06) (logid:87edf8d7) ===START=== 10.80.0.6
>> -- GET
>> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
>>
>> 2018-12-13 08:44:06,665 DEBUG [c.c.a.ApiServer]
>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) CIDRs
>> from which account 'Acct[26e597f2-b5ca-11e8-a619-c8cbb8cb15cd-admin]' is
>> allowed to perform API calls: 0.0.0.0/0,::/0
>> 2018-12-13 08:44:06,693 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:7e0c4dc9) Add job-1343
>> into job monitoring
>> 2018-12-13 08:44:06,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) submit
>> async job-1343, details: AsyncJobVO {id:1343, userId: 2, accountId: 2,
>> instanceType: VirtualMachine, instanceId: 8, cmd:
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 220777304233416, completeMsid: null,
>> lastUpdated: null, lastPolled: null, created: null}
>> 2018-12-13 08:44:06,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Executing
>> AsyncJobVO {id:1343, userId: 2, accountId: 2, instanceType:
>> VirtualMachine, instanceId: 8, cmd:
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 220777304233416, completeMsid: null,
>> lastUpdated: null, lastPolled: null, created: null}
>> 2018-12-13 08:44:06,705 DEBUG [c.c.a.ApiServlet]
>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7)
>> ===END=== 10.80.0.6 -- GET
>> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
>>
>> 2018-12-13 08:44:06,745 DEBUG [c.c.n.NetworkModelImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Service SecurityGroup is not supported in the network id=205
>> 2018-12-13 08:44:06,752 DEBUG [c.c.n.NetworkModelImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Service SecurityGroup is not supported in the network id=205
>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> DeploymentPlanner allocation algorithm: null
>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Trying to allocate a host and storage pools from dc:1,
>> pod:null,cluster:null, requested cpu: 8000, requested ram: 8594128896
>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Is ROOT volume READY (pool already allocated)?: Yes
>> 2018-12-13 08:44:06,783 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Deploy avoids pods: [], clusters: [], hosts: []
>> 2018-12-13 08:44:06,784 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> This VM has last host_id specified, trying to choose the same host: 10
>> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Host: 10 has cpu capability (cpu:12, speed:3000) to support requested
>> CPU: 4 and requested speed: 2000
>> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Checking if host: 10 has enough capacity for requested CPU: 8000 and
>> requested RAM: 8594128896 , cpuOverprovisioningFactor: 1.0
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Hosts's actual total CPU: 36000 and CPU after applying overprovisioning:
>> 36000
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> We need to allocate to the last host again, so checking if there is
>> enough reserved capacity
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Reserved CPU: 8000 , Requested CPU: 8000
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Reserved RAM: 8594128896 , Requested RAM: 8594128896
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Host has enough CPU and RAM available
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> STATS: Can alloc CPU from host: 10, used: 4500, reserved: 8000, actual
>> total: 36000, total with overprovisioning: 36000; requested
>> cpu:8000,alloc_from_last_host?:true ,considerReservedCapacity?: true
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> STATS: Can alloc MEM from host: 10, used: 2415919104, reserved:
>> 8594128896, total: 49554284544; requested mem:
>> 8594128896,alloc_from_last_host?:true ,considerReservedCapacity?: true
>> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> The last host of this VM is UP and has enough capacity
>> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Now checking for suitable pools under zone: 1, pod: 1, cluster: 1
>> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Checking suitable pools for volume (Id, Type): (8,ROOT)
>> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Volume has pool already allocated, checking if pool can be reused,
>> poolId: 1
>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Planner need not allocate a pool for this volume since its READY
>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Checking suitable pools for volume (Id, Type): (12,DATADISK)
>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Volume has pool already allocated, checking if pool can be reused,
>> poolId: 1
>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Planner need not allocate a pool for this volume since its READY
>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Checking suitable pools for volume (Id, Type): (17,DATADISK)
>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Volume has pool already allocated, checking if pool can be reused,
>> poolId: 1
>> 2018-12-13 08:44:06,819 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Planner need not allocate a pool for this volume since its READY
>> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Checking suitable pools for volume (Id, Type): (58,DATADISK)
>> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Volume has pool already allocated, checking if pool can be reused,
>> poolId: 1
>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Planner need not allocate a pool for this volume since its READY
>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Trying to find a potenial host and associated storage pools from the
>> suitable host/pool lists for this VM
>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Checking if host: 10 can access any suitable storage pool for volume:
>> DATADISK
>> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Host: 10 can access pool: 1
>> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Checking if host: 10 can access any suitable storage pool for volume:
>> DATADISK
>> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Host: 10 can access pool: 1
>> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Checking if host: 10 can access any suitable storage pool for volume:
>> DATADISK
>> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Host: 10 can access pool: 1
>> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Checking if host: 10 can access any suitable storage pool for volume: ROOT
>> 2018-12-13 08:44:06,831 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Host: 10 can access pool: 1
>> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Found a potential host id: 10 name: cshp143 and associated storage pools
>> for this VM
>> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Returning Deployment Destination:
>> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
>> : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(10)-Storage()]
>> 2018-12-13 08:44:06,856 DEBUG [c.c.u.d.T.Transaction]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Rolling back the transaction: Time = 3 Name = API-Job-Executor-8; called
>> by
>> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:233-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
>>
>> 2018-12-13 08:44:06,861 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Unable to schedule async job for command com.cloud.vm.VmWorkStart,
>> unexpected exception.
>> javax.persistence.EntityExistsException: Entity already exists:
>> at
>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
>>
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
>>
>> at
>> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
>> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
>>
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>>
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>>
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>>
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:233)
>>
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>>
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>>
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>
>> at
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>>
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>>
>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
>> at
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>>
>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>> at
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>>
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>>
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by:
>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>> Duplicate entry '1344' for key 'PRIMARY'
>> at
>> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> ... 59 more
>> 2018-12-13 08:44:06,876 DEBUG [c.c.u.d.T.Transaction]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Rolling back the transaction: Time = 2 Name = API-Job-Executor-8; called
>> by
>> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:245-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
>>
>> 2018-12-13 08:44:06,880 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
>> Unable to schedule async job for command com.cloud.vm.VmWorkStart,
>> unexpected exception.
>> javax.persistence.EntityExistsException: Entity already exists:
>> at
>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
>>
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
>>
>> at
>> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
>> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
>>
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>>
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>>
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>>
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
>>
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>>
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>>
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>
>> at
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>>
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>>
>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
>> at
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>>
>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>> at
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>>
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>>
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by:
>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>> Duplicate entry '1345' for key 'PRIMARY'
>> at
>> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> ... 59 more
>> 2018-12-13 08:44:06,889 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Unexpected
>> exception while executing
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
>> com.cloud.utils.exception.CloudRuntimeException: Unable to schedule
>> async job for command com.cloud.vm.VmWorkStart, unexpected exception.
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
>>
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>>
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>>
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>>
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
>>
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>>
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>>
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>
>> at
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>>
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>>
>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
>> at
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>>
>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>> at
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>>
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>>
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>
>> at java.lang.Thread.run(Thread.java:748)
>> 2018-12-13 08:44:06,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Complete
>> async job-1343, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
>> to schedule async job for command com.cloud.vm.VmWorkStart, unexpected
>> exception."}
>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Publish
>> async job-1343 complete on message bus
>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up jobs
>> related to job-1343
>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Update db
>> status for job-1343
>> 2018-12-13 08:44:06,895 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up jobs
>> joined with job-1343 and disjoin all subjobs created from job- 1343
>> 2018-12-13 08:44:06,899 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Done
>> executing org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
>> for job-1343
>> 2018-12-13 08:44:06,899 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Remove
>> job-1343 from job monitoring
>> 2018-12-13 08:44:07,056 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) Begin cleanup
>> expired async-jobs
>> 2018-12-13 08:44:07,062 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) End cleanup
>> expired async-jobs
>>
>> This problem has already appeared to me other times (in other
>> circumstances) and I have reported it but I have not found a solution.
>>
>> Can you help me identify the problem?
>>
>
>
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.
Re: URGENT: Unable to schedule async job for command
com.cloud.vm.VmWorkStart, unexpected exception
Posted by Rene Moser <ma...@renemoser.net>.
Hi
I think you hit https://github.com/apache/cloudstack/issues/2880
Is this already 4.11.2?
We have also seen this in our lab, could not really reproduce it, seems
to be a race condition.
Regards
René
On 12/13/18 9:01 AM, Ugo Vasi wrote:
> Hi all,
> I'm trying to reboot a vm after the host it ran on crashed and restarted
> from the HA system. All the VMs running on the rebooted host were
> restarted on other hosts except one.
> In the web interface and using cloudmonkey I get this message:
> "Unable to schedule async job for command com.cloud.vm.VmWorkStart,
> unexpected exception."
>
> In the management-server.log file there would seem to be a problem when
> creating an element that is duplicated (Duplicate entry 'xxxx' for key
> 'PRIMARY'):
>
> 2018-12-13 08:44:06,659 DEBUG [c.c.a.ApiServlet]
> (qtp1096283470-445:ctx-6c065e06) (logid:87edf8d7) ===START=== 10.80.0.6
> -- GET
> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
>
> 2018-12-13 08:44:06,665 DEBUG [c.c.a.ApiServer]
> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) CIDRs
> from which account 'Acct[26e597f2-b5ca-11e8-a619-c8cbb8cb15cd-admin]' is
> allowed to perform API calls: 0.0.0.0/0,::/0
> 2018-12-13 08:44:06,693 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:7e0c4dc9) Add job-1343
> into job monitoring
> 2018-12-13 08:44:06,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) submit
> async job-1343, details: AsyncJobVO {id:1343, userId: 2, accountId: 2,
> instanceType: VirtualMachine, instanceId: 8, cmd:
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null}
> 2018-12-13 08:44:06,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Executing
> AsyncJobVO {id:1343, userId: 2, accountId: 2, instanceType:
> VirtualMachine, instanceId: 8, cmd:
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null}
> 2018-12-13 08:44:06,705 DEBUG [c.c.a.ApiServlet]
> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7)
> ===END=== 10.80.0.6 -- GET
> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
>
> 2018-12-13 08:44:06,745 DEBUG [c.c.n.NetworkModelImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Service SecurityGroup is not supported in the network id=205
> 2018-12-13 08:44:06,752 DEBUG [c.c.n.NetworkModelImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Service SecurityGroup is not supported in the network id=205
> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> DeploymentPlanner allocation algorithm: null
> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Trying to allocate a host and storage pools from dc:1,
> pod:null,cluster:null, requested cpu: 8000, requested ram: 8594128896
> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Is ROOT volume READY (pool already allocated)?: Yes
> 2018-12-13 08:44:06,783 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Deploy avoids pods: [], clusters: [], hosts: []
> 2018-12-13 08:44:06,784 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> This VM has last host_id specified, trying to choose the same host: 10
> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Host: 10 has cpu capability (cpu:12, speed:3000) to support requested
> CPU: 4 and requested speed: 2000
> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Checking if host: 10 has enough capacity for requested CPU: 8000 and
> requested RAM: 8594128896 , cpuOverprovisioningFactor: 1.0
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Hosts's actual total CPU: 36000 and CPU after applying overprovisioning:
> 36000
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> We need to allocate to the last host again, so checking if there is
> enough reserved capacity
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Reserved CPU: 8000 , Requested CPU: 8000
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Reserved RAM: 8594128896 , Requested RAM: 8594128896
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Host has enough CPU and RAM available
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> STATS: Can alloc CPU from host: 10, used: 4500, reserved: 8000, actual
> total: 36000, total with overprovisioning: 36000; requested
> cpu:8000,alloc_from_last_host?:true ,considerReservedCapacity?: true
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> STATS: Can alloc MEM from host: 10, used: 2415919104, reserved:
> 8594128896, total: 49554284544; requested mem:
> 8594128896,alloc_from_last_host?:true ,considerReservedCapacity?: true
> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> The last host of this VM is UP and has enough capacity
> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Now checking for suitable pools under zone: 1, pod: 1, cluster: 1
> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Checking suitable pools for volume (Id, Type): (8,ROOT)
> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Volume has pool already allocated, checking if pool can be reused,
> poolId: 1
> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Planner need not allocate a pool for this volume since its READY
> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Checking suitable pools for volume (Id, Type): (12,DATADISK)
> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Volume has pool already allocated, checking if pool can be reused,
> poolId: 1
> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Planner need not allocate a pool for this volume since its READY
> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Checking suitable pools for volume (Id, Type): (17,DATADISK)
> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Volume has pool already allocated, checking if pool can be reused,
> poolId: 1
> 2018-12-13 08:44:06,819 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Planner need not allocate a pool for this volume since its READY
> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Checking suitable pools for volume (Id, Type): (58,DATADISK)
> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Volume has pool already allocated, checking if pool can be reused,
> poolId: 1
> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Planner need not allocate a pool for this volume since its READY
> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Trying to find a potenial host and associated storage pools from the
> suitable host/pool lists for this VM
> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Checking if host: 10 can access any suitable storage pool for volume:
> DATADISK
> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Host: 10 can access pool: 1
> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Checking if host: 10 can access any suitable storage pool for volume:
> DATADISK
> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Host: 10 can access pool: 1
> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Checking if host: 10 can access any suitable storage pool for volume:
> DATADISK
> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Host: 10 can access pool: 1
> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Checking if host: 10 can access any suitable storage pool for volume: ROOT
> 2018-12-13 08:44:06,831 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Host: 10 can access pool: 1
> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Found a potential host id: 10 name: cshp143 and associated storage pools
> for this VM
> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Returning Deployment Destination:
> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
> : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(10)-Storage()]
> 2018-12-13 08:44:06,856 DEBUG [c.c.u.d.T.Transaction]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Rolling back the transaction: Time = 3 Name = API-Job-Executor-8; called
> by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:233-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
>
> 2018-12-13 08:44:06,861 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Unable to schedule async job for command com.cloud.vm.VmWorkStart,
> unexpected exception.
> javax.persistence.EntityExistsException: Entity already exists:
> at
> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
>
> at
> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:233)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>
> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
> at
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Duplicate entry '1344' for key 'PRIMARY'
> at
> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> ... 59 more
> 2018-12-13 08:44:06,876 DEBUG [c.c.u.d.T.Transaction]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Rolling back the transaction: Time = 2 Name = API-Job-Executor-8; called
> by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:245-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
>
> 2018-12-13 08:44:06,880 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5) (logid:6e9a71c5)
> Unable to schedule async job for command com.cloud.vm.VmWorkStart,
> unexpected exception.
> javax.persistence.EntityExistsException: Entity already exists:
> at
> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
>
> at
> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>
> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
> at
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Duplicate entry '1345' for key 'PRIMARY'
> at
> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> ... 59 more
> 2018-12-13 08:44:06,889 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
> com.cloud.utils.exception.CloudRuntimeException: Unable to schedule
> async job for command com.cloud.vm.VmWorkStart, unexpected exception.
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>
> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
> at
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> 2018-12-13 08:44:06,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Complete
> async job-1343, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
> to schedule async job for command com.cloud.vm.VmWorkStart, unexpected
> exception."}
> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Publish
> async job-1343 complete on message bus
> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up jobs
> related to job-1343
> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Update db
> status for job-1343
> 2018-12-13 08:44:06,895 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up jobs
> joined with job-1343 and disjoin all subjobs created from job- 1343
> 2018-12-13 08:44:06,899 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Done
> executing org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
> for job-1343
> 2018-12-13 08:44:06,899 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Remove
> job-1343 from job monitoring
> 2018-12-13 08:44:07,056 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) Begin cleanup
> expired async-jobs
> 2018-12-13 08:44:07,062 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) End cleanup
> expired async-jobs
>
> This problem has already appeared to me other times (in other
> circumstances) and I have reported it but I have not found a solution.
>
> Can you help me identify the problem?
>
Re: URGENT: Unable to schedule async job for command
com.cloud.vm.VmWorkStart, unexpected exception
Posted by Ugo Vasi <ug...@procne.it.INVALID>.
Hi Andrija, hi all,
this morning I stumbled on the same problem some time ago... after that
catastrophic experience I reinstalled the infrastructure that was
unusable. The current new infrastructure is updated to version 4.11.2.0,
again on Ubuntu 16.04, KVM as hypervisor and Advanced network.
I tried restarting a network and I got an error "Unable to schedule
async job for command com.cloud.vm.VmWorkStart, unexpected exception.".
From what I see in the logs (of which I will report below an excerpt)
The procedure created the new router but at the time of starting it,
something went wrong and the process was canceled.
The exception refers to a "Duplicate entry '1083' for key 'PRIMARY'", so
I analyzed the database dump and I found a record in the "vm_work_job"
table with the primary key id=1083:
INSERT INTO `vm_work_job` VALUES ...
(1081,'Prepare','Instance',45),(1083,'Starting','Instance',45),(1085,'Prepare','Instance',45),(1087,'Starting','Instance',45),(1089,'Starting','Instance',45),(1092,'Starting','Instance',27);
On mysql error log I don't find any error message.
From management-server.log:
2019-02-22 08:26:46,678 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(qtp1096283470-458:ctx-b9d2a3ee ctx-cf002500) (logid:b267e0bd) submit
async job-1082, details: AsyncJobVO {id:1082, userId: 2, accountId: 2,
instanceType: None, instanceId: null, cmd:
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd,
cmdInfo:
{"makeredundant":"false","cleanup":"true","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2891","id":"bc0cc3a2-e21f-4904-a826-04477b1805c0","ctxDetails":"{\"interface
com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}","ctxAccountId":"2","uuid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","cmdEventType":"NETWORK.RESTART","_":"1550820391247"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 220777304233416, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null}
2019-02-22 08:26:46,721 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082) (logid:ddff8687) Add job-1082
into job monitoring
2019-02-22 08:26:46,727 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082) (logid:f7040f44) Executing
AsyncJobVO {id:1082, userId: 2, accountId: 2, instanceType: None,
instanceId: null, cmd:
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd,
cmdInfo:
{"makeredundant":"false","cleanup":"true","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2891","id":"bc0cc3a2-e21f-4904-a826-04477b1805c0","ctxDetails":"{\"interface
com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}","ctxAccountId":"2","uuid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","cmdEventType":"NETWORK.RESTART","_":"1550820391247"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 220777304233416, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null}
2019-02-22 08:26:46,793 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Restarting network 206...
2019-02-22 08:26:46,811 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Performing rolling restart of routers of network Ntwk[206|Guest|8]
2019-02-22 08:26:46,950 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Asking VirtualRouter to implemenet Ntwk[206|Guest|8]
2019-02-22 08:26:47,145 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Allocating the VR with id=47 in datacenter
com.cloud.dc.DataCenterVO$$EnhancerByCGLIB$$2e90024a@1 with the
hypervisor type KVM
2019-02-22 08:26:47,161 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-12:null) (logid:) SeqA 2-159198: Processing Seq
2-159198: { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2019-02-22 08:26:47,163 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-12:null) (logid:) SeqA 2-159198: Sending Seq
2-159198: { Ans: , MgmtId: 220777304233416, via: 2, Ver: v1, Flags:
100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2019-02-22 08:26:47,281 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Adding nic for Virtual Router in Guest network Ntwk[206|Guest|8]
2019-02-22 08:26:47,281 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Adding nic for Virtual Router in Control network
2019-02-22 08:26:47,296 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Found existing network configuration for offering [Network Offering
[3-Control-System-Control-Network]: Ntwk[202|Control|3]
2019-02-22 08:26:47,296 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Releasing lock for Acct[4c03aaf5-ffa3-11e8-bcb0-c8cbb8cb15cd-system]
2019-02-22 08:26:47,299 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Adding nic for Virtual Router in Public network
2019-02-22 08:26:47,313 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Found existing network configuration for offering [Network Offering
[1-Public-System-Public-Network]: Ntwk[200|Public|1]
2019-02-22 08:26:47,313 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Releasing lock for Acct[4c03aaf5-ffa3-11e8-bcb0-c8cbb8cb15cd-system]
2019-02-22 08:26:47,318 INFO [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Use same MAC as previous RvR, the MAC is 1e:00:0c:00:00:14
2019-02-22 08:26:47,322 DEBUG [c.c.v.VirtualMachineManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Allocating entries for VM: VM[DomainRouter|r-47-VM]
2019-02-22 08:26:47,378 DEBUG [c.c.v.VirtualMachineManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Allocating nics for VM[DomainRouter|r-47-VM]
2019-02-22 08:26:47,411 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Allocating nic for vm VM[DomainRouter|r-47-VM] in network
Ntwk[206|Guest|8] with requested profile
NicProfile[0-0-null-10.10.10.1-vlan://13
2019-02-22 08:26:47,460 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Service SecurityGroup is not supported in the network id=206
2019-02-22 08:26:47,482 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Allocating nic for vm VM[DomainRouter|r-47-VM] in network
Ntwk[202|Control|3] with requested profile null
2019-02-22 08:26:47,492 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Allocating nic for vm VM[DomainRouter|r-47-VM] in network
Ntwk[200|Public|1] with requested profile
NicProfile[0-0-null-193.239.54.68-vlan://untagged
2019-02-22 08:26:47,503 DEBUG [c.c.v.VirtualMachineManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Allocating disks for VM[DomainRouter|r-47-VM]
2019-02-22 08:26:47,594 DEBUG [c.c.v.VirtualMachineManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Allocation completed for VM: VM[DomainRouter|r-47-VM]
2019-02-22 08:26:47,903 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Starting router VM[DomainRouter|r-47-VM]
2019-02-22 08:26:48,149 DEBUG [c.c.u.d.T.Transaction]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Rolling back the transaction: Time = 14 Name = API-Job-Executor-1;
called by
-TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-TransactionLegacy.close:657-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4464-VirtualMachineManagerImpl.advanceStart:899-NetworkHelperImpl.start:276-NetworkHelperImpl.startVirtualRouter:355-NetworkHelperImpl.startRouters:340-RouterDeploymentDefinition.deployVirtualRouter:205
2019-02-22 08:26:48,234 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082 ctx-d2b2e42b) (logid:f7040f44)
Unable to schedule async job for command com.cloud.vm.VmWorkStart,
unexpected exception.
javax.persistence.EntityExistsException: Entity already exists:
at
com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
...
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Duplicate entry '1083' for key 'PRIMARY'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
... 64 more
2019-02-22 08:26:48,262 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082) (logid:f7040f44) Unexpected
exception while executing
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
at
com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4464)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
...
2019-02-22 08:26:48,294 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082) (logid:f7040f44) Complete
async job-1082, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
to schedule async job for command com.cloud.vm.VmWorkStart, unexpected
exception."}
2019-02-22 08:26:48,295 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082) (logid:f7040f44) Publish
async job-1082 complete on message bus
2019-02-22 08:26:48,295 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082) (logid:f7040f44) Wake up jobs
related to job-1082
2019-02-22 08:26:48,295 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082) (logid:f7040f44) Update db
status for job-1082
2019-02-22 08:26:48,296 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082) (logid:f7040f44) Wake up jobs
joined with job-1082 and disjoin all subjobs created from job- 1082
2019-02-22 08:26:48,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082) (logid:f7040f44) Done
executing
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for
job-1082
2019-02-22 08:26:48,357 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-1:ctx-bd6a5e5b job-1082) (logid:f7040f44) Remove
job-1082 from job monitoring
Il 14/12/18 11:05, Andrija Panic ha scritto:
> That's funny - does restarting mgmg helps ? I assume you restored full
> cloud DB "property"...
>
> On Fri, Dec 14, 2018, 09:42 Ugo Vasi <ugo.vasi@procne.it wrote:
>
>> Hi Andrija,
>> I restored database but now I can't see any instance, host or system vm.
>> On dashboard I see a real situation statistics but I can't see any data,
>> neither via webUI nor via cloudmonkey... I verify existence of the
>> instances/hosts data on cloud's database tables.
>>
>>
>>
>> Il 13/12/18 17:22, Andrija Panic ha scritto:
>>> Done it once (8h old DB)...make sure to kill any new VMs that were
>> created
>>> in that period of time (and volumes/templates on the end storage).
>>>
>>> You might run into situation that some VMs are i.e. powered on during
>> that
>>> last 8h, so DB state for that VM will not be in sync with reality, but
>> that
>>> should be synced shorty after.
>>>
>>> I.e. hope that a very few changes/creations of any resources have
>> happened
>>> during that period - and expect some stuff will need to be manually
>>> synced/fixed/repeated - i.e. user created PF rule on VR, but now you have
>>> restored DB so - that might be a challenge to fix without pissing off
>> users
>>> :)
>>>
>>> Fingers crossed!
>>>
>>> On Thu, Dec 13, 2018, 14:33 Ugo Vasi <ugo.vasi@procne.it.invalid wrote:
>>>
>>>> Hi all,
>>>> is there anyone who tried to restore the database a day ago and
>>>> restarted cloudstack manager?
>>>>
>>>> The two situations differ for the status of the VM / VR and I do not
>>>> know what to expect and bearing in mind that the problem could recur.
>>>>
>>>> The statistics don't interest me.
>>>>
>>>>
>>>> Il 13/12/18 10:39, Ugo Vasi ha scritto:
>>>>> We have verified that the problem is not tied to a vm but it also
>>>>> concerns the other VMs that are stopped. Trying to restart them gives
>>>>> the same error message.
>>>>>
>>>>> Restoring the previous day's database which problems might imply?
>>>>>
>>>>>
>>>>> Il 13/12/18 09:01, Ugo Vasi ha scritto:
>>>>>> Hi all,
>>>>>> I'm trying to reboot a vm after the host it ran on crashed and
>>>>>> restarted from the HA system. All the VMs running on the rebooted
>>>>>> host were restarted on other hosts except one.
>>>>>> In the web interface and using cloudmonkey I get this message:
>>>>>> "Unable to schedule async job for command com.cloud.vm.VmWorkStart,
>>>>>> unexpected exception."
>>>>>>
>>>>>> In the management-server.log file there would seem to be a problem
>>>>>> when creating an element that is duplicated (Duplicate entry 'xxxx'
>>>>>> for key 'PRIMARY'):
>>>>>>
>>>>>> 2018-12-13 08:44:06,659 DEBUG [c.c.a.ApiServlet]
>>>>>> (qtp1096283470-445:ctx-6c065e06) (logid:87edf8d7) ===START===
>>>>>> 10.80.0.6 -- GET
>>>>>>
>> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
>>>>>> 2018-12-13 08:44:06,665 DEBUG [c.c.a.ApiServer]
>>>>>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) CIDRs
>>>>>> from which account 'Acct[26e597f2-b5ca-11e8-a619-c8cbb8cb15cd-admin]'
>>>>>> is allowed to perform API calls: 0.0.0.0/0,::/0
>>>>>> 2018-12-13 08:44:06,693 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:7e0c4dc9) Add
>>>>>> job-1343 into job monitoring
>>>>>> 2018-12-13 08:44:06,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) submit
>>>>>> async job-1343, details: AsyncJobVO {id:1343, userId: 2, accountId:
>>>>>> 2, instanceType: VirtualMachine, instanceId: 8, cmd:
>>>>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
>>>>>> cmdInfo:
>>>>>>
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>>>>
>> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>>>>> result: null, initMsid: 220777304233416, completeMsid: null,
>>>>>> lastUpdated: null, lastPolled: null, created: null}
>>>>>> 2018-12-13 08:44:06,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Executing
>>>>>> AsyncJobVO {id:1343, userId: 2, accountId: 2, instanceType:
>>>>>> VirtualMachine, instanceId: 8, cmd:
>>>>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
>>>>>> cmdInfo:
>>>>>>
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>>>>
>> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>>>>> result: null, initMsid: 220777304233416, completeMsid: null,
>>>>>> lastUpdated: null, lastPolled: null, created: null}
>>>>>> 2018-12-13 08:44:06,705 DEBUG [c.c.a.ApiServlet]
>>>>>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7)
>>>>>> ===END=== 10.80.0.6 -- GET
>>>>>>
>> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
>>>>>> 2018-12-13 08:44:06,745 DEBUG [c.c.n.NetworkModelImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Service SecurityGroup is not supported in the
>>>>>> network id=205
>>>>>> 2018-12-13 08:44:06,752 DEBUG [c.c.n.NetworkModelImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Service SecurityGroup is not supported in the
>>>>>> network id=205
>>>>>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) DeploymentPlanner allocation algorithm: null
>>>>>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Trying to allocate a host and storage pools from
>>>>>> dc:1, pod:null,cluster:null, requested cpu: 8000, requested ram:
>>>>>> 8594128896
>>>>>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Is ROOT volume READY (pool already allocated)?: Yes
>>>>>> 2018-12-13 08:44:06,783 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Deploy avoids pods: [], clusters: [], hosts: []
>>>>>> 2018-12-13 08:44:06,784 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) This VM has last host_id specified, trying to choose
>>>>>> the same host: 10
>>>>>> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Host: 10 has cpu capability (cpu:12, speed:3000) to
>>>>>> support requested CPU: 4 and requested speed: 2000
>>>>>> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Checking if host: 10 has enough capacity for
>>>>>> requested CPU: 8000 and requested RAM: 8594128896 ,
>>>>>> cpuOverprovisioningFactor: 1.0
>>>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Hosts's actual total CPU: 36000 and CPU after
>>>>>> applying overprovisioning: 36000
>>>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) We need to allocate to the last host again, so
>>>>>> checking if there is enough reserved capacity
>>>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Reserved CPU: 8000 , Requested CPU: 8000
>>>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Reserved RAM: 8594128896 , Requested RAM: 8594128896
>>>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Host has enough CPU and RAM available
>>>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) STATS: Can alloc CPU from host: 10, used: 4500,
>>>>>> reserved: 8000, actual total: 36000, total with overprovisioning:
>>>>>> 36000; requested cpu:8000,alloc_from_last_host?:true
>>>>>> ,considerReservedCapacity?: true
>>>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) STATS: Can alloc MEM from host: 10, used:
>>>>>> 2415919104, reserved: 8594128896, total: 49554284544; requested mem:
>>>>>> 8594128896,alloc_from_last_host?:true ,considerReservedCapacity?: true
>>>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) The last host of this VM is UP and has enough
>> capacity
>>>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Now checking for suitable pools under zone: 1, pod:
>>>>>> 1, cluster: 1
>>>>>> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
>> (8,ROOT)
>>>>>> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>>>>>> can be reused, poolId: 1
>>>>>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>>>>>> since its READY
>>>>>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
>>>>>> (12,DATADISK)
>>>>>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>>>>>> can be reused, poolId: 1
>>>>>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>>>>>> since its READY
>>>>>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
>>>>>> (17,DATADISK)
>>>>>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>>>>>> can be reused, poolId: 1
>>>>>> 2018-12-13 08:44:06,819 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>>>>>> since its READY
>>>>>> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
>>>>>> (58,DATADISK)
>>>>>> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>>>>>> can be reused, poolId: 1
>>>>>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>>>>>> since its READY
>>>>>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Trying to find a potenial host and associated
>>>>>> storage pools from the suitable host/pool lists for this VM
>>>>>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>>>>>> pool for volume: DATADISK
>>>>>> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Host: 10 can access pool: 1
>>>>>> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>>>>>> pool for volume: DATADISK
>>>>>> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Host: 10 can access pool: 1
>>>>>> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>>>>>> pool for volume: DATADISK
>>>>>> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Host: 10 can access pool: 1
>>>>>> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>>>>>> pool for volume: ROOT
>>>>>> 2018-12-13 08:44:06,831 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Host: 10 can access pool: 1
>>>>>> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Found a potential host id: 10 name: cshp143 and
>>>>>> associated storage pools for this VM
>>>>>> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Returning Deployment Destination:
>>>>>>
>> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
>>>>>> : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(10)-Storage()]
>>>>>> 2018-12-13 08:44:06,856 DEBUG [c.c.u.d.T.Transaction]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Rolling back the transaction: Time = 3 Name =
>>>>>> API-Job-Executor-8; called by
>>>>>>
>> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:233-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
>>>>>> 2018-12-13 08:44:06,861 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Unable to schedule async job for command
>>>>>> com.cloud.vm.VmWorkStart, unexpected exception.
>>>>>> javax.persistence.EntityExistsException: Entity already exists:
>>>>>> at
>>>>>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
>>>>>> at
>>>>>> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
>>>>>> at
>> com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>>>>> at
>> com.cloud.utils.db.Transaction.execute(Transaction.java:47)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
>>>>>> at
>>>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>>>>>> at
>>>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>>>>>> at
>>>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:233)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>>>>>> at
>>>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>>>>>> at
>>>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>>>>>> at
>>>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>> at
>>>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>> at
>>>>>>
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>>>> at
>>>>>>
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>>>> at
>>>>>>
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>>>>>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown
>> Source)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>>>>>> at
>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>>>>>> at
>>>>>>
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>>>>>> at
>>>>>>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>> at
>>>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>> at
>>>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>> Caused by:
>>>>>>
>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>>>>>> Duplicate entry '1344' for key 'PRIMARY'
>>>>>> at
>>>>>> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown
>> Source)
>>>>>> at
>>>>>>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at
>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>>>>> ... 59 more
>>>>>> 2018-12-13 08:44:06,876 DEBUG [c.c.u.d.T.Transaction]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Rolling back the transaction: Time = 2 Name =
>>>>>> API-Job-Executor-8; called by
>>>>>>
>> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:245-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
>>>>>> 2018-12-13 08:44:06,880 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>>>> (logid:6e9a71c5) Unable to schedule async job for command
>>>>>> com.cloud.vm.VmWorkStart, unexpected exception.
>>>>>> javax.persistence.EntityExistsException: Entity already exists:
>>>>>> at
>>>>>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
>>>>>> at
>>>>>> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
>>>>>> at
>> com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>>>>> at
>> com.cloud.utils.db.Transaction.execute(Transaction.java:47)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
>>>>>> at
>>>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>>>>>> at
>>>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>>>>>> at
>>>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>>>>>> at
>>>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>>>>>> at
>>>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>>>>>> at
>>>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>> at
>>>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>> at
>>>>>>
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>>>> at
>>>>>>
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>>>> at
>>>>>>
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>>>>>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown
>> Source)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>>>>>> at
>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>>>>>> at
>>>>>>
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>>>>>> at
>>>>>>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>> at
>>>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>> at
>>>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>> Caused by:
>>>>>>
>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>>>>>> Duplicate entry '1345' for key 'PRIMARY'
>>>>>> at
>>>>>> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown
>> Source)
>>>>>> at
>>>>>>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at
>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>>>>> ... 59 more
>>>>>> 2018-12-13 08:44:06,889 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5)
>>>>>> Unexpected exception while executing
>>>>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
>>>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to schedule
>>>>>> async job for command com.cloud.vm.VmWorkStart, unexpected exception.
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
>>>>>> at
>>>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>>>>>> at
>>>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>>>>>> at
>>>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>>>>>> at
>>>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>>>>>> at
>>>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>>>>>> at
>>>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>> at
>>>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>> at
>>>>>>
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>>>> at
>>>>>>
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>>>> at
>>>>>>
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>>>>>> at
>>>>>>
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>>>>>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown
>> Source)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>>>>>> at
>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>>>>>> at
>>>>>>
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>>>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>>>>>> at
>>>>>>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>> at
>>>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>> at
>>>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>> 2018-12-13 08:44:06,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Complete
>>>>>> async job-1343, jobStatus: FAILED, resultCode: 530, result:
>>>>>>
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
>>>>>> to schedule async job for command com.cloud.vm.VmWorkStart,
>>>>>> unexpected exception."}
>>>>>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Publish
>>>>>> async job-1343 complete on message bus
>>>>>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
>>>>>> jobs related to job-1343
>>>>>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Update db
>>>>>> status for job-1343
>>>>>> 2018-12-13 08:44:06,895 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
>>>>>> jobs joined with job-1343 and disjoin all subjobs created from job-
>> 1343
>>>>>> 2018-12-13 08:44:06,899 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Done
>>>>>> executing
>>>>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin for
>>>>>> job-1343
>>>>>> 2018-12-13 08:44:06,899 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Remove
>>>>>> job-1343 from job monitoring
>>>>>> 2018-12-13 08:44:07,056 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) Begin cleanup
>>>>>> expired async-jobs
>>>>>> 2018-12-13 08:44:07,062 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) End cleanup
>>>>>> expired async-jobs
>>>>>>
>>>>>> This problem has already appeared to me other times (in other
>>>>>> circumstances) and I have reported it but I have not found a solution.
>>>>>>
>>>>>> Can you help me identify the problem?
>>>>>>
>>>> --
>>>>
>>>> *Ugo Vasi* / System Administrator
>>>> ugo.vasi@procne.it <ma...@procne.it>
>>>>
>>>>
>>>>
>>>>
>>>> *Procne S.r.l.*
>>>> +39 0432 486 523
>>>> via Cotonificio, 45
>>>> 33010 Tavagnacco (UD)
>>>> www.procne.it <http://www.procne.it/>
>>>>
>>>>
>>>> Le informazioni contenute nella presente comunicazione ed i relativi
>>>> allegati possono essere riservate e sono, comunque, destinate
>>>> esclusivamente alle persone od alla Società sopraindicati. La
>>>> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
>>>> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
>>>> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
>>>> "Codice in materia di protezione dei dati personali". Se avete ricevuto
>>>> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
>>>> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
>>>> info@procne.it <ma...@procne.it>.
>>>>
>>>>
>>>
>>>
>>
>> --
>>
>> *Ugo Vasi* / System Administrator
>> ugo.vasi@procne.it <ma...@procne.it>
>>
>>
>>
>>
>> *Procne S.r.l.*
>> +39 0432 486 523
>> via Cotonificio, 45
>> 33010 Tavagnacco (UD)
>> www.procne.it <http://www.procne.it/>
>>
>>
>> Le informazioni contenute nella presente comunicazione ed i relativi
>> allegati possono essere riservate e sono, comunque, destinate
>> esclusivamente alle persone od alla Società sopraindicati. La
>> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
>> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
>> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
>> "Codice in materia di protezione dei dati personali". Se avete ricevuto
>> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
>> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
>> info@procne.it <ma...@procne.it>.
>>
>>
>
>
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.
Re: URGENT: Unable to schedule async job for command
com.cloud.vm.VmWorkStart, unexpected exception
Posted by Andrija Panic <an...@gmail.com>.
That's funny - does restarting mgmg helps ? I assume you restored full
cloud DB "property"...
On Fri, Dec 14, 2018, 09:42 Ugo Vasi <ugo.vasi@procne.it wrote:
> Hi Andrija,
> I restored database but now I can't see any instance, host or system vm.
> On dashboard I see a real situation statistics but I can't see any data,
> neither via webUI nor via cloudmonkey... I verify existence of the
> instances/hosts data on cloud's database tables.
>
>
>
> Il 13/12/18 17:22, Andrija Panic ha scritto:
> > Done it once (8h old DB)...make sure to kill any new VMs that were
> created
> > in that period of time (and volumes/templates on the end storage).
> >
> > You might run into situation that some VMs are i.e. powered on during
> that
> > last 8h, so DB state for that VM will not be in sync with reality, but
> that
> > should be synced shorty after.
> >
> > I.e. hope that a very few changes/creations of any resources have
> happened
> > during that period - and expect some stuff will need to be manually
> > synced/fixed/repeated - i.e. user created PF rule on VR, but now you have
> > restored DB so - that might be a challenge to fix without pissing off
> users
> > :)
> >
> > Fingers crossed!
> >
> > On Thu, Dec 13, 2018, 14:33 Ugo Vasi <ugo.vasi@procne.it.invalid wrote:
> >
> >> Hi all,
> >> is there anyone who tried to restore the database a day ago and
> >> restarted cloudstack manager?
> >>
> >> The two situations differ for the status of the VM / VR and I do not
> >> know what to expect and bearing in mind that the problem could recur.
> >>
> >> The statistics don't interest me.
> >>
> >>
> >> Il 13/12/18 10:39, Ugo Vasi ha scritto:
> >>> We have verified that the problem is not tied to a vm but it also
> >>> concerns the other VMs that are stopped. Trying to restart them gives
> >>> the same error message.
> >>>
> >>> Restoring the previous day's database which problems might imply?
> >>>
> >>>
> >>> Il 13/12/18 09:01, Ugo Vasi ha scritto:
> >>>> Hi all,
> >>>> I'm trying to reboot a vm after the host it ran on crashed and
> >>>> restarted from the HA system. All the VMs running on the rebooted
> >>>> host were restarted on other hosts except one.
> >>>> In the web interface and using cloudmonkey I get this message:
> >>>> "Unable to schedule async job for command com.cloud.vm.VmWorkStart,
> >>>> unexpected exception."
> >>>>
> >>>> In the management-server.log file there would seem to be a problem
> >>>> when creating an element that is duplicated (Duplicate entry 'xxxx'
> >>>> for key 'PRIMARY'):
> >>>>
> >>>> 2018-12-13 08:44:06,659 DEBUG [c.c.a.ApiServlet]
> >>>> (qtp1096283470-445:ctx-6c065e06) (logid:87edf8d7) ===START===
> >>>> 10.80.0.6 -- GET
> >>>>
> >>
> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
> >>>> 2018-12-13 08:44:06,665 DEBUG [c.c.a.ApiServer]
> >>>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) CIDRs
> >>>> from which account 'Acct[26e597f2-b5ca-11e8-a619-c8cbb8cb15cd-admin]'
> >>>> is allowed to perform API calls: 0.0.0.0/0,::/0
> >>>> 2018-12-13 08:44:06,693 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:7e0c4dc9) Add
> >>>> job-1343 into job monitoring
> >>>> 2018-12-13 08:44:06,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) submit
> >>>> async job-1343, details: AsyncJobVO {id:1343, userId: 2, accountId:
> >>>> 2, instanceType: VirtualMachine, instanceId: 8, cmd:
> >>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
> >>>> cmdInfo:
> >>>>
> >>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
> >>
> >>
> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
> >>
> >>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >>>> result: null, initMsid: 220777304233416, completeMsid: null,
> >>>> lastUpdated: null, lastPolled: null, created: null}
> >>>> 2018-12-13 08:44:06,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Executing
> >>>> AsyncJobVO {id:1343, userId: 2, accountId: 2, instanceType:
> >>>> VirtualMachine, instanceId: 8, cmd:
> >>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
> >>>> cmdInfo:
> >>>>
> >>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
> >>
> >>
> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
> >>
> >>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >>>> result: null, initMsid: 220777304233416, completeMsid: null,
> >>>> lastUpdated: null, lastPolled: null, created: null}
> >>>> 2018-12-13 08:44:06,705 DEBUG [c.c.a.ApiServlet]
> >>>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7)
> >>>> ===END=== 10.80.0.6 -- GET
> >>>>
> >>
> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
> >>>> 2018-12-13 08:44:06,745 DEBUG [c.c.n.NetworkModelImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Service SecurityGroup is not supported in the
> >>>> network id=205
> >>>> 2018-12-13 08:44:06,752 DEBUG [c.c.n.NetworkModelImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Service SecurityGroup is not supported in the
> >>>> network id=205
> >>>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) DeploymentPlanner allocation algorithm: null
> >>>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Trying to allocate a host and storage pools from
> >>>> dc:1, pod:null,cluster:null, requested cpu: 8000, requested ram:
> >>>> 8594128896
> >>>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Is ROOT volume READY (pool already allocated)?: Yes
> >>>> 2018-12-13 08:44:06,783 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Deploy avoids pods: [], clusters: [], hosts: []
> >>>> 2018-12-13 08:44:06,784 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) This VM has last host_id specified, trying to choose
> >>>> the same host: 10
> >>>> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Host: 10 has cpu capability (cpu:12, speed:3000) to
> >>>> support requested CPU: 4 and requested speed: 2000
> >>>> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Checking if host: 10 has enough capacity for
> >>>> requested CPU: 8000 and requested RAM: 8594128896 ,
> >>>> cpuOverprovisioningFactor: 1.0
> >>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Hosts's actual total CPU: 36000 and CPU after
> >>>> applying overprovisioning: 36000
> >>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) We need to allocate to the last host again, so
> >>>> checking if there is enough reserved capacity
> >>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Reserved CPU: 8000 , Requested CPU: 8000
> >>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Reserved RAM: 8594128896 , Requested RAM: 8594128896
> >>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Host has enough CPU and RAM available
> >>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) STATS: Can alloc CPU from host: 10, used: 4500,
> >>>> reserved: 8000, actual total: 36000, total with overprovisioning:
> >>>> 36000; requested cpu:8000,alloc_from_last_host?:true
> >>>> ,considerReservedCapacity?: true
> >>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) STATS: Can alloc MEM from host: 10, used:
> >>>> 2415919104, reserved: 8594128896, total: 49554284544; requested mem:
> >>>> 8594128896,alloc_from_last_host?:true ,considerReservedCapacity?: true
> >>>> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) The last host of this VM is UP and has enough
> capacity
> >>>> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Now checking for suitable pools under zone: 1, pod:
> >>>> 1, cluster: 1
> >>>> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
> (8,ROOT)
> >>>> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> >>>> can be reused, poolId: 1
> >>>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> >>>> since its READY
> >>>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
> >>>> (12,DATADISK)
> >>>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> >>>> can be reused, poolId: 1
> >>>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> >>>> since its READY
> >>>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
> >>>> (17,DATADISK)
> >>>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> >>>> can be reused, poolId: 1
> >>>> 2018-12-13 08:44:06,819 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> >>>> since its READY
> >>>> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
> >>>> (58,DATADISK)
> >>>> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> >>>> can be reused, poolId: 1
> >>>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> >>>> since its READY
> >>>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Trying to find a potenial host and associated
> >>>> storage pools from the suitable host/pool lists for this VM
> >>>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> >>>> pool for volume: DATADISK
> >>>> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Host: 10 can access pool: 1
> >>>> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> >>>> pool for volume: DATADISK
> >>>> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Host: 10 can access pool: 1
> >>>> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> >>>> pool for volume: DATADISK
> >>>> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Host: 10 can access pool: 1
> >>>> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> >>>> pool for volume: ROOT
> >>>> 2018-12-13 08:44:06,831 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Host: 10 can access pool: 1
> >>>> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Found a potential host id: 10 name: cshp143 and
> >>>> associated storage pools for this VM
> >>>> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Returning Deployment Destination:
> >>>>
> >>
> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
> >>
> >>>> : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(10)-Storage()]
> >>>> 2018-12-13 08:44:06,856 DEBUG [c.c.u.d.T.Transaction]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Rolling back the transaction: Time = 3 Name =
> >>>> API-Job-Executor-8; called by
> >>>>
> >>
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:233-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
> >>>> 2018-12-13 08:44:06,861 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Unable to schedule async job for command
> >>>> com.cloud.vm.VmWorkStart, unexpected exception.
> >>>> javax.persistence.EntityExistsException: Entity already exists:
> >>>> at
> >>>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
> >>>> at
> >>>> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
> >>>> at
> com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> >>>> at
> com.cloud.utils.db.Transaction.execute(Transaction.java:47)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
> >>>> at
> >>>>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
> >>>> at
> >>>>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
> >>>> at
> >>>>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:233)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
> >>>> at
> >>>>
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
> >>>> at
> >>>>
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> >>>> at
> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>>> at
> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> at java.lang.reflect.Method.invoke(Method.java:498)
> >>>> at
> >>>>
> >>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >>>> at
> >>>>
> >>
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >>>> at
> >>>>
> >>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >>>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown
> Source)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
> >>>> at
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> >>>> at
> >>>>
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> >>>> at
> >>>>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >>>> at
> >>>>
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>>> at
> >>>>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>>> at java.lang.Thread.run(Thread.java:748)
> >>>> Caused by:
> >>>>
> >>
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> >>>> Duplicate entry '1344' for key 'PRIMARY'
> >>>> at
> >>>> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown
> Source)
> >>>> at
> >>>>
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> >>>> at
> >>>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> >>>> ... 59 more
> >>>> 2018-12-13 08:44:06,876 DEBUG [c.c.u.d.T.Transaction]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Rolling back the transaction: Time = 2 Name =
> >>>> API-Job-Executor-8; called by
> >>>>
> >>
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:245-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
> >>>> 2018-12-13 08:44:06,880 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >>>> (logid:6e9a71c5) Unable to schedule async job for command
> >>>> com.cloud.vm.VmWorkStart, unexpected exception.
> >>>> javax.persistence.EntityExistsException: Entity already exists:
> >>>> at
> >>>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
> >>>> at
> >>>> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
> >>>> at
> com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> >>>> at
> com.cloud.utils.db.Transaction.execute(Transaction.java:47)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
> >>>> at
> >>>>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
> >>>> at
> >>>>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
> >>>> at
> >>>>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
> >>>> at
> >>>>
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
> >>>> at
> >>>>
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> >>>> at
> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>>> at
> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> at java.lang.reflect.Method.invoke(Method.java:498)
> >>>> at
> >>>>
> >>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >>>> at
> >>>>
> >>
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >>>> at
> >>>>
> >>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >>>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown
> Source)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
> >>>> at
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> >>>> at
> >>>>
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> >>>> at
> >>>>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >>>> at
> >>>>
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>>> at
> >>>>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>>> at java.lang.Thread.run(Thread.java:748)
> >>>> Caused by:
> >>>>
> >>
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> >>>> Duplicate entry '1345' for key 'PRIMARY'
> >>>> at
> >>>> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown
> Source)
> >>>> at
> >>>>
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> >>>> at
> >>>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> >>>> ... 59 more
> >>>> 2018-12-13 08:44:06,889 ERROR [c.c.a.ApiAsyncJobDispatcher]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5)
> >>>> Unexpected exception while executing
> >>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
> >>>> com.cloud.utils.exception.CloudRuntimeException: Unable to schedule
> >>>> async job for command com.cloud.vm.VmWorkStart, unexpected exception.
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
> >>>> at
> >>>>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
> >>>> at
> >>>>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
> >>>> at
> >>>>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
> >>>> at
> >>>>
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
> >>>> at
> >>>>
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> >>>> at
> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>>> at
> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> at java.lang.reflect.Method.invoke(Method.java:498)
> >>>> at
> >>>>
> >>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >>>> at
> >>>>
> >>
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >>>> at
> >>>>
> >>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> >>>> at
> >>>>
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >>>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown
> Source)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
> >>>> at
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> >>>> at
> >>>>
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >>>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> >>>> at
> >>>>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >>>> at
> >>>>
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>>> at
> >>>>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>>> at java.lang.Thread.run(Thread.java:748)
> >>>> 2018-12-13 08:44:06,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Complete
> >>>> async job-1343, jobStatus: FAILED, resultCode: 530, result:
> >>>>
> >>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
> >>
> >>>> to schedule async job for command com.cloud.vm.VmWorkStart,
> >>>> unexpected exception."}
> >>>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Publish
> >>>> async job-1343 complete on message bus
> >>>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
> >>>> jobs related to job-1343
> >>>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Update db
> >>>> status for job-1343
> >>>> 2018-12-13 08:44:06,895 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
> >>>> jobs joined with job-1343 and disjoin all subjobs created from job-
> 1343
> >>>> 2018-12-13 08:44:06,899 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Done
> >>>> executing
> >>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin for
> >>>> job-1343
> >>>> 2018-12-13 08:44:06,899 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Remove
> >>>> job-1343 from job monitoring
> >>>> 2018-12-13 08:44:07,056 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) Begin cleanup
> >>>> expired async-jobs
> >>>> 2018-12-13 08:44:07,062 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) End cleanup
> >>>> expired async-jobs
> >>>>
> >>>> This problem has already appeared to me other times (in other
> >>>> circumstances) and I have reported it but I have not found a solution.
> >>>>
> >>>> Can you help me identify the problem?
> >>>>
> >>>
> >>
> >> --
> >>
> >> *Ugo Vasi* / System Administrator
> >> ugo.vasi@procne.it <ma...@procne.it>
> >>
> >>
> >>
> >>
> >> *Procne S.r.l.*
> >> +39 0432 486 523
> >> via Cotonificio, 45
> >> 33010 Tavagnacco (UD)
> >> www.procne.it <http://www.procne.it/>
> >>
> >>
> >> Le informazioni contenute nella presente comunicazione ed i relativi
> >> allegati possono essere riservate e sono, comunque, destinate
> >> esclusivamente alle persone od alla Società sopraindicati. La
> >> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
> >> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
> >> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
> >> "Codice in materia di protezione dei dati personali". Se avete ricevuto
> >> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
> >> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
> >> info@procne.it <ma...@procne.it>.
> >>
> >>
> >
> >
> >
>
>
> --
>
> *Ugo Vasi* / System Administrator
> ugo.vasi@procne.it <ma...@procne.it>
>
>
>
>
> *Procne S.r.l.*
> +39 0432 486 523
> via Cotonificio, 45
> 33010 Tavagnacco (UD)
> www.procne.it <http://www.procne.it/>
>
>
> Le informazioni contenute nella presente comunicazione ed i relativi
> allegati possono essere riservate e sono, comunque, destinate
> esclusivamente alle persone od alla Società sopraindicati. La
> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
> "Codice in materia di protezione dei dati personali". Se avete ricevuto
> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
> info@procne.it <ma...@procne.it>.
>
>
Re: URGENT: Unable to schedule async job for command
com.cloud.vm.VmWorkStart, unexpected exception
Posted by Ugo Vasi <ug...@procne.it.INVALID>.
Hi Andrija,
I restored database but now I can't see any instance, host or system vm.
On dashboard I see a real situation statistics but I can't see any data,
neither via webUI nor via cloudmonkey... I verify existence of the
instances/hosts data on cloud's database tables.
Il 13/12/18 17:22, Andrija Panic ha scritto:
> Done it once (8h old DB)...make sure to kill any new VMs that were created
> in that period of time (and volumes/templates on the end storage).
>
> You might run into situation that some VMs are i.e. powered on during that
> last 8h, so DB state for that VM will not be in sync with reality, but that
> should be synced shorty after.
>
> I.e. hope that a very few changes/creations of any resources have happened
> during that period - and expect some stuff will need to be manually
> synced/fixed/repeated - i.e. user created PF rule on VR, but now you have
> restored DB so - that might be a challenge to fix without pissing off users
> :)
>
> Fingers crossed!
>
> On Thu, Dec 13, 2018, 14:33 Ugo Vasi <ugo.vasi@procne.it.invalid wrote:
>
>> Hi all,
>> is there anyone who tried to restore the database a day ago and
>> restarted cloudstack manager?
>>
>> The two situations differ for the status of the VM / VR and I do not
>> know what to expect and bearing in mind that the problem could recur.
>>
>> The statistics don't interest me.
>>
>>
>> Il 13/12/18 10:39, Ugo Vasi ha scritto:
>>> We have verified that the problem is not tied to a vm but it also
>>> concerns the other VMs that are stopped. Trying to restart them gives
>>> the same error message.
>>>
>>> Restoring the previous day's database which problems might imply?
>>>
>>>
>>> Il 13/12/18 09:01, Ugo Vasi ha scritto:
>>>> Hi all,
>>>> I'm trying to reboot a vm after the host it ran on crashed and
>>>> restarted from the HA system. All the VMs running on the rebooted
>>>> host were restarted on other hosts except one.
>>>> In the web interface and using cloudmonkey I get this message:
>>>> "Unable to schedule async job for command com.cloud.vm.VmWorkStart,
>>>> unexpected exception."
>>>>
>>>> In the management-server.log file there would seem to be a problem
>>>> when creating an element that is duplicated (Duplicate entry 'xxxx'
>>>> for key 'PRIMARY'):
>>>>
>>>> 2018-12-13 08:44:06,659 DEBUG [c.c.a.ApiServlet]
>>>> (qtp1096283470-445:ctx-6c065e06) (logid:87edf8d7) ===START===
>>>> 10.80.0.6 -- GET
>>>>
>> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
>>>> 2018-12-13 08:44:06,665 DEBUG [c.c.a.ApiServer]
>>>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) CIDRs
>>>> from which account 'Acct[26e597f2-b5ca-11e8-a619-c8cbb8cb15cd-admin]'
>>>> is allowed to perform API calls: 0.0.0.0/0,::/0
>>>> 2018-12-13 08:44:06,693 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:7e0c4dc9) Add
>>>> job-1343 into job monitoring
>>>> 2018-12-13 08:44:06,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) submit
>>>> async job-1343, details: AsyncJobVO {id:1343, userId: 2, accountId:
>>>> 2, instanceType: VirtualMachine, instanceId: 8, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
>>>> cmdInfo:
>>>>
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>>
>> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>>
>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>>> result: null, initMsid: 220777304233416, completeMsid: null,
>>>> lastUpdated: null, lastPolled: null, created: null}
>>>> 2018-12-13 08:44:06,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Executing
>>>> AsyncJobVO {id:1343, userId: 2, accountId: 2, instanceType:
>>>> VirtualMachine, instanceId: 8, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
>>>> cmdInfo:
>>>>
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>>
>> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>>
>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>>> result: null, initMsid: 220777304233416, completeMsid: null,
>>>> lastUpdated: null, lastPolled: null, created: null}
>>>> 2018-12-13 08:44:06,705 DEBUG [c.c.a.ApiServlet]
>>>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7)
>>>> ===END=== 10.80.0.6 -- GET
>>>>
>> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
>>>> 2018-12-13 08:44:06,745 DEBUG [c.c.n.NetworkModelImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Service SecurityGroup is not supported in the
>>>> network id=205
>>>> 2018-12-13 08:44:06,752 DEBUG [c.c.n.NetworkModelImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Service SecurityGroup is not supported in the
>>>> network id=205
>>>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) DeploymentPlanner allocation algorithm: null
>>>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Trying to allocate a host and storage pools from
>>>> dc:1, pod:null,cluster:null, requested cpu: 8000, requested ram:
>>>> 8594128896
>>>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Is ROOT volume READY (pool already allocated)?: Yes
>>>> 2018-12-13 08:44:06,783 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Deploy avoids pods: [], clusters: [], hosts: []
>>>> 2018-12-13 08:44:06,784 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) This VM has last host_id specified, trying to choose
>>>> the same host: 10
>>>> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Host: 10 has cpu capability (cpu:12, speed:3000) to
>>>> support requested CPU: 4 and requested speed: 2000
>>>> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Checking if host: 10 has enough capacity for
>>>> requested CPU: 8000 and requested RAM: 8594128896 ,
>>>> cpuOverprovisioningFactor: 1.0
>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Hosts's actual total CPU: 36000 and CPU after
>>>> applying overprovisioning: 36000
>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) We need to allocate to the last host again, so
>>>> checking if there is enough reserved capacity
>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Reserved CPU: 8000 , Requested CPU: 8000
>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Reserved RAM: 8594128896 , Requested RAM: 8594128896
>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Host has enough CPU and RAM available
>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) STATS: Can alloc CPU from host: 10, used: 4500,
>>>> reserved: 8000, actual total: 36000, total with overprovisioning:
>>>> 36000; requested cpu:8000,alloc_from_last_host?:true
>>>> ,considerReservedCapacity?: true
>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) STATS: Can alloc MEM from host: 10, used:
>>>> 2415919104, reserved: 8594128896, total: 49554284544; requested mem:
>>>> 8594128896,alloc_from_last_host?:true ,considerReservedCapacity?: true
>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) The last host of this VM is UP and has enough capacity
>>>> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Now checking for suitable pools under zone: 1, pod:
>>>> 1, cluster: 1
>>>> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type): (8,ROOT)
>>>> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>>>> can be reused, poolId: 1
>>>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>>>> since its READY
>>>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
>>>> (12,DATADISK)
>>>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>>>> can be reused, poolId: 1
>>>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>>>> since its READY
>>>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
>>>> (17,DATADISK)
>>>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>>>> can be reused, poolId: 1
>>>> 2018-12-13 08:44:06,819 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>>>> since its READY
>>>> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
>>>> (58,DATADISK)
>>>> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>>>> can be reused, poolId: 1
>>>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>>>> since its READY
>>>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Trying to find a potenial host and associated
>>>> storage pools from the suitable host/pool lists for this VM
>>>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>>>> pool for volume: DATADISK
>>>> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Host: 10 can access pool: 1
>>>> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>>>> pool for volume: DATADISK
>>>> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Host: 10 can access pool: 1
>>>> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>>>> pool for volume: DATADISK
>>>> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Host: 10 can access pool: 1
>>>> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>>>> pool for volume: ROOT
>>>> 2018-12-13 08:44:06,831 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Host: 10 can access pool: 1
>>>> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Found a potential host id: 10 name: cshp143 and
>>>> associated storage pools for this VM
>>>> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Returning Deployment Destination:
>>>>
>> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
>>
>>>> : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(10)-Storage()]
>>>> 2018-12-13 08:44:06,856 DEBUG [c.c.u.d.T.Transaction]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Rolling back the transaction: Time = 3 Name =
>>>> API-Job-Executor-8; called by
>>>>
>> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:233-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
>>>> 2018-12-13 08:44:06,861 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Unable to schedule async job for command
>>>> com.cloud.vm.VmWorkStart, unexpected exception.
>>>> javax.persistence.EntityExistsException: Entity already exists:
>>>> at
>>>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
>>>> at
>>>> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
>>>> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>>> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
>>>> at
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>>>> at
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>>>> at
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>>>> at
>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:233)
>>>> at
>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>>>> at
>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>>>> at
>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at
>>>>
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>>> at
>>>>
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>> at
>>>>
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>> at
>>>>
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>>>> at
>>>>
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>>>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
>>>> at
>>>>
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>>>> at
>>>>
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by:
>>>>
>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>>>> Duplicate entry '1344' for key 'PRIMARY'
>>>> at
>>>> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
>>>> at
>>>>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>> at
>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>>> ... 59 more
>>>> 2018-12-13 08:44:06,876 DEBUG [c.c.u.d.T.Transaction]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Rolling back the transaction: Time = 2 Name =
>>>> API-Job-Executor-8; called by
>>>>
>> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:245-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
>>>> 2018-12-13 08:44:06,880 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>>>> (logid:6e9a71c5) Unable to schedule async job for command
>>>> com.cloud.vm.VmWorkStart, unexpected exception.
>>>> javax.persistence.EntityExistsException: Entity already exists:
>>>> at
>>>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
>>>> at
>>>> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
>>>> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>>> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
>>>> at
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>>>> at
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>>>> at
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>>>> at
>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
>>>> at
>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>>>> at
>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>>>> at
>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at
>>>>
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>>> at
>>>>
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>> at
>>>>
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>> at
>>>>
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>>>> at
>>>>
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>>>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
>>>> at
>>>>
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>>>> at
>>>>
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by:
>>>>
>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>>>> Duplicate entry '1345' for key 'PRIMARY'
>>>> at
>>>> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
>>>> at
>>>>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>> at
>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>>> ... 59 more
>>>> 2018-12-13 08:44:06,889 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5)
>>>> Unexpected exception while executing
>>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to schedule
>>>> async job for command com.cloud.vm.VmWorkStart, unexpected exception.
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
>>>> at
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>>>> at
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>>>> at
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>>>> at
>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
>>>> at
>>>>
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>>>> at
>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>>>> at
>>>>
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at
>>>>
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>>> at
>>>>
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>> at
>>>>
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>>>> at
>>>>
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>>>> at
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>>>> at
>>>>
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>>>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
>>>> at
>>>>
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>>>> at
>>>>
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>> at
>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> 2018-12-13 08:44:06,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Complete
>>>> async job-1343, jobStatus: FAILED, resultCode: 530, result:
>>>>
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
>>
>>>> to schedule async job for command com.cloud.vm.VmWorkStart,
>>>> unexpected exception."}
>>>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Publish
>>>> async job-1343 complete on message bus
>>>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
>>>> jobs related to job-1343
>>>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Update db
>>>> status for job-1343
>>>> 2018-12-13 08:44:06,895 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
>>>> jobs joined with job-1343 and disjoin all subjobs created from job- 1343
>>>> 2018-12-13 08:44:06,899 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Done
>>>> executing
>>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin for
>>>> job-1343
>>>> 2018-12-13 08:44:06,899 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Remove
>>>> job-1343 from job monitoring
>>>> 2018-12-13 08:44:07,056 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) Begin cleanup
>>>> expired async-jobs
>>>> 2018-12-13 08:44:07,062 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) End cleanup
>>>> expired async-jobs
>>>>
>>>> This problem has already appeared to me other times (in other
>>>> circumstances) and I have reported it but I have not found a solution.
>>>>
>>>> Can you help me identify the problem?
>>>>
>>>
>>
>> --
>>
>> *Ugo Vasi* / System Administrator
>> ugo.vasi@procne.it <ma...@procne.it>
>>
>>
>>
>>
>> *Procne S.r.l.*
>> +39 0432 486 523
>> via Cotonificio, 45
>> 33010 Tavagnacco (UD)
>> www.procne.it <http://www.procne.it/>
>>
>>
>> Le informazioni contenute nella presente comunicazione ed i relativi
>> allegati possono essere riservate e sono, comunque, destinate
>> esclusivamente alle persone od alla Società sopraindicati. La
>> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
>> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
>> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
>> "Codice in materia di protezione dei dati personali". Se avete ricevuto
>> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
>> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
>> info@procne.it <ma...@procne.it>.
>>
>>
>
>
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.
Re: URGENT: Unable to schedule async job for command
com.cloud.vm.VmWorkStart, unexpected exception
Posted by Andrija Panic <an...@gmail.com>.
Done it once (8h old DB)...make sure to kill any new VMs that were created
in that period of time (and volumes/templates on the end storage).
You might run into situation that some VMs are i.e. powered on during that
last 8h, so DB state for that VM will not be in sync with reality, but that
should be synced shorty after.
I.e. hope that a very few changes/creations of any resources have happened
during that period - and expect some stuff will need to be manually
synced/fixed/repeated - i.e. user created PF rule on VR, but now you have
restored DB so - that might be a challenge to fix without pissing off users
:)
Fingers crossed!
On Thu, Dec 13, 2018, 14:33 Ugo Vasi <ugo.vasi@procne.it.invalid wrote:
> Hi all,
> is there anyone who tried to restore the database a day ago and
> restarted cloudstack manager?
>
> The two situations differ for the status of the VM / VR and I do not
> know what to expect and bearing in mind that the problem could recur.
>
> The statistics don't interest me.
>
>
> Il 13/12/18 10:39, Ugo Vasi ha scritto:
> > We have verified that the problem is not tied to a vm but it also
> > concerns the other VMs that are stopped. Trying to restart them gives
> > the same error message.
> >
> > Restoring the previous day's database which problems might imply?
> >
> >
> > Il 13/12/18 09:01, Ugo Vasi ha scritto:
> >> Hi all,
> >> I'm trying to reboot a vm after the host it ran on crashed and
> >> restarted from the HA system. All the VMs running on the rebooted
> >> host were restarted on other hosts except one.
> >> In the web interface and using cloudmonkey I get this message:
> >> "Unable to schedule async job for command com.cloud.vm.VmWorkStart,
> >> unexpected exception."
> >>
> >> In the management-server.log file there would seem to be a problem
> >> when creating an element that is duplicated (Duplicate entry 'xxxx'
> >> for key 'PRIMARY'):
> >>
> >> 2018-12-13 08:44:06,659 DEBUG [c.c.a.ApiServlet]
> >> (qtp1096283470-445:ctx-6c065e06) (logid:87edf8d7) ===START===
> >> 10.80.0.6 -- GET
> >>
> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
> >> 2018-12-13 08:44:06,665 DEBUG [c.c.a.ApiServer]
> >> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) CIDRs
> >> from which account 'Acct[26e597f2-b5ca-11e8-a619-c8cbb8cb15cd-admin]'
> >> is allowed to perform API calls: 0.0.0.0/0,::/0
> >> 2018-12-13 08:44:06,693 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:7e0c4dc9) Add
> >> job-1343 into job monitoring
> >> 2018-12-13 08:44:06,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) submit
> >> async job-1343, details: AsyncJobVO {id:1343, userId: 2, accountId:
> >> 2, instanceType: VirtualMachine, instanceId: 8, cmd:
> >> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
> >> cmdInfo:
> >>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>
> >>
> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>
> >> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >> result: null, initMsid: 220777304233416, completeMsid: null,
> >> lastUpdated: null, lastPolled: null, created: null}
> >> 2018-12-13 08:44:06,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Executing
> >> AsyncJobVO {id:1343, userId: 2, accountId: 2, instanceType:
> >> VirtualMachine, instanceId: 8, cmd:
> >> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
> >> cmdInfo:
> >>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>
> >>
> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>
> >> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >> result: null, initMsid: 220777304233416, completeMsid: null,
> >> lastUpdated: null, lastPolled: null, created: null}
> >> 2018-12-13 08:44:06,705 DEBUG [c.c.a.ApiServlet]
> >> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7)
> >> ===END=== 10.80.0.6 -- GET
> >>
> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
> >> 2018-12-13 08:44:06,745 DEBUG [c.c.n.NetworkModelImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Service SecurityGroup is not supported in the
> >> network id=205
> >> 2018-12-13 08:44:06,752 DEBUG [c.c.n.NetworkModelImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Service SecurityGroup is not supported in the
> >> network id=205
> >> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) DeploymentPlanner allocation algorithm: null
> >> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Trying to allocate a host and storage pools from
> >> dc:1, pod:null,cluster:null, requested cpu: 8000, requested ram:
> >> 8594128896
> >> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Is ROOT volume READY (pool already allocated)?: Yes
> >> 2018-12-13 08:44:06,783 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Deploy avoids pods: [], clusters: [], hosts: []
> >> 2018-12-13 08:44:06,784 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) This VM has last host_id specified, trying to choose
> >> the same host: 10
> >> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Host: 10 has cpu capability (cpu:12, speed:3000) to
> >> support requested CPU: 4 and requested speed: 2000
> >> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Checking if host: 10 has enough capacity for
> >> requested CPU: 8000 and requested RAM: 8594128896 ,
> >> cpuOverprovisioningFactor: 1.0
> >> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Hosts's actual total CPU: 36000 and CPU after
> >> applying overprovisioning: 36000
> >> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) We need to allocate to the last host again, so
> >> checking if there is enough reserved capacity
> >> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Reserved CPU: 8000 , Requested CPU: 8000
> >> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Reserved RAM: 8594128896 , Requested RAM: 8594128896
> >> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Host has enough CPU and RAM available
> >> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) STATS: Can alloc CPU from host: 10, used: 4500,
> >> reserved: 8000, actual total: 36000, total with overprovisioning:
> >> 36000; requested cpu:8000,alloc_from_last_host?:true
> >> ,considerReservedCapacity?: true
> >> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) STATS: Can alloc MEM from host: 10, used:
> >> 2415919104, reserved: 8594128896, total: 49554284544; requested mem:
> >> 8594128896,alloc_from_last_host?:true ,considerReservedCapacity?: true
> >> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) The last host of this VM is UP and has enough capacity
> >> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Now checking for suitable pools under zone: 1, pod:
> >> 1, cluster: 1
> >> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type): (8,ROOT)
> >> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> >> can be reused, poolId: 1
> >> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> >> since its READY
> >> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
> >> (12,DATADISK)
> >> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> >> can be reused, poolId: 1
> >> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> >> since its READY
> >> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
> >> (17,DATADISK)
> >> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> >> can be reused, poolId: 1
> >> 2018-12-13 08:44:06,819 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> >> since its READY
> >> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
> >> (58,DATADISK)
> >> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> >> can be reused, poolId: 1
> >> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> >> since its READY
> >> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Trying to find a potenial host and associated
> >> storage pools from the suitable host/pool lists for this VM
> >> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> >> pool for volume: DATADISK
> >> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Host: 10 can access pool: 1
> >> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> >> pool for volume: DATADISK
> >> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Host: 10 can access pool: 1
> >> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> >> pool for volume: DATADISK
> >> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Host: 10 can access pool: 1
> >> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> >> pool for volume: ROOT
> >> 2018-12-13 08:44:06,831 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Host: 10 can access pool: 1
> >> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Found a potential host id: 10 name: cshp143 and
> >> associated storage pools for this VM
> >> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Returning Deployment Destination:
> >>
> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
>
> >> : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(10)-Storage()]
> >> 2018-12-13 08:44:06,856 DEBUG [c.c.u.d.T.Transaction]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Rolling back the transaction: Time = 3 Name =
> >> API-Job-Executor-8; called by
> >>
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:233-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
> >> 2018-12-13 08:44:06,861 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Unable to schedule async job for command
> >> com.cloud.vm.VmWorkStart, unexpected exception.
> >> javax.persistence.EntityExistsException: Entity already exists:
> >> at
> >> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
> >> at
> >> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
> >> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> >> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
> >> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
> >> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
> >> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
> >> at
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:233)
> >> at
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
> >> at
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
> >> at
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:498)
> >> at
> >>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >> at
> >>
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >> at
> >>
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >> at
> >>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> >> at
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
> >> at
> >>
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> >> at
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> >> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> >> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >> at java.lang.Thread.run(Thread.java:748)
> >> Caused by:
> >>
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> >> Duplicate entry '1344' for key 'PRIMARY'
> >> at
> >> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
> >> at
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> >> at
> >> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> >> ... 59 more
> >> 2018-12-13 08:44:06,876 DEBUG [c.c.u.d.T.Transaction]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Rolling back the transaction: Time = 2 Name =
> >> API-Job-Executor-8; called by
> >>
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:245-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
> >> 2018-12-13 08:44:06,880 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> >> (logid:6e9a71c5) Unable to schedule async job for command
> >> com.cloud.vm.VmWorkStart, unexpected exception.
> >> javax.persistence.EntityExistsException: Entity already exists:
> >> at
> >> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
> >> at
> >> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
> >> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> >> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
> >> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
> >> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
> >> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
> >> at
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
> >> at
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
> >> at
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
> >> at
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:498)
> >> at
> >>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >> at
> >>
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >> at
> >>
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >> at
> >>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> >> at
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
> >> at
> >>
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> >> at
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> >> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> >> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >> at java.lang.Thread.run(Thread.java:748)
> >> Caused by:
> >>
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> >> Duplicate entry '1345' for key 'PRIMARY'
> >> at
> >> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
> >> at
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> >> at
> >> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> >> ... 59 more
> >> 2018-12-13 08:44:06,889 ERROR [c.c.a.ApiAsyncJobDispatcher]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5)
> >> Unexpected exception while executing
> >> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
> >> com.cloud.utils.exception.CloudRuntimeException: Unable to schedule
> >> async job for command com.cloud.vm.VmWorkStart, unexpected exception.
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
> >> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
> >> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
> >> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
> >> at
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
> >> at
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
> >> at
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
> >> at
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:498)
> >> at
> >>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >> at
> >>
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >> at
> >>
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >> at
> >>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> >> at
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
> >> at
> >>
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> >> at
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> >> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> >> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >> at java.lang.Thread.run(Thread.java:748)
> >> 2018-12-13 08:44:06,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Complete
> >> async job-1343, jobStatus: FAILED, resultCode: 530, result:
> >>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
>
> >> to schedule async job for command com.cloud.vm.VmWorkStart,
> >> unexpected exception."}
> >> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Publish
> >> async job-1343 complete on message bus
> >> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
> >> jobs related to job-1343
> >> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Update db
> >> status for job-1343
> >> 2018-12-13 08:44:06,895 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
> >> jobs joined with job-1343 and disjoin all subjobs created from job- 1343
> >> 2018-12-13 08:44:06,899 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Done
> >> executing
> >> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin for
> >> job-1343
> >> 2018-12-13 08:44:06,899 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Remove
> >> job-1343 from job monitoring
> >> 2018-12-13 08:44:07,056 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) Begin cleanup
> >> expired async-jobs
> >> 2018-12-13 08:44:07,062 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) End cleanup
> >> expired async-jobs
> >>
> >> This problem has already appeared to me other times (in other
> >> circumstances) and I have reported it but I have not found a solution.
> >>
> >> Can you help me identify the problem?
> >>
> >
> >
>
>
> --
>
> *Ugo Vasi* / System Administrator
> ugo.vasi@procne.it <ma...@procne.it>
>
>
>
>
> *Procne S.r.l.*
> +39 0432 486 523
> via Cotonificio, 45
> 33010 Tavagnacco (UD)
> www.procne.it <http://www.procne.it/>
>
>
> Le informazioni contenute nella presente comunicazione ed i relativi
> allegati possono essere riservate e sono, comunque, destinate
> esclusivamente alle persone od alla Società sopraindicati. La
> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
> "Codice in materia di protezione dei dati personali". Se avete ricevuto
> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
> info@procne.it <ma...@procne.it>.
>
>
Re: URGENT: Unable to schedule async job for command
com.cloud.vm.VmWorkStart, unexpected exception
Posted by Ugo Vasi <ug...@procne.it.INVALID>.
Hi all,
is there anyone who tried to restore the database a day ago and
restarted cloudstack manager?
The two situations differ for the status of the VM / VR and I do not
know what to expect and bearing in mind that the problem could recur.
The statistics don't interest me.
Il 13/12/18 10:39, Ugo Vasi ha scritto:
> We have verified that the problem is not tied to a vm but it also
> concerns the other VMs that are stopped. Trying to restart them gives
> the same error message.
>
> Restoring the previous day's database which problems might imply?
>
>
> Il 13/12/18 09:01, Ugo Vasi ha scritto:
>> Hi all,
>> I'm trying to reboot a vm after the host it ran on crashed and
>> restarted from the HA system. All the VMs running on the rebooted
>> host were restarted on other hosts except one.
>> In the web interface and using cloudmonkey I get this message:
>> "Unable to schedule async job for command com.cloud.vm.VmWorkStart,
>> unexpected exception."
>>
>> In the management-server.log file there would seem to be a problem
>> when creating an element that is duplicated (Duplicate entry 'xxxx'
>> for key 'PRIMARY'):
>>
>> 2018-12-13 08:44:06,659 DEBUG [c.c.a.ApiServlet]
>> (qtp1096283470-445:ctx-6c065e06) (logid:87edf8d7) ===START===
>> 10.80.0.6 -- GET
>> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
>> 2018-12-13 08:44:06,665 DEBUG [c.c.a.ApiServer]
>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) CIDRs
>> from which account 'Acct[26e597f2-b5ca-11e8-a619-c8cbb8cb15cd-admin]'
>> is allowed to perform API calls: 0.0.0.0/0,::/0
>> 2018-12-13 08:44:06,693 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:7e0c4dc9) Add
>> job-1343 into job monitoring
>> 2018-12-13 08:44:06,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) submit
>> async job-1343, details: AsyncJobVO {id:1343, userId: 2, accountId:
>> 2, instanceType: VirtualMachine, instanceId: 8, cmd:
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
>> cmdInfo:
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 220777304233416, completeMsid: null,
>> lastUpdated: null, lastPolled: null, created: null}
>> 2018-12-13 08:44:06,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Executing
>> AsyncJobVO {id:1343, userId: 2, accountId: 2, instanceType:
>> VirtualMachine, instanceId: 8, cmd:
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin,
>> cmdInfo:
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
>> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 220777304233416, completeMsid: null,
>> lastUpdated: null, lastPolled: null, created: null}
>> 2018-12-13 08:44:06,705 DEBUG [c.c.a.ApiServlet]
>> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7)
>> ===END=== 10.80.0.6 -- GET
>> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
>> 2018-12-13 08:44:06,745 DEBUG [c.c.n.NetworkModelImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Service SecurityGroup is not supported in the
>> network id=205
>> 2018-12-13 08:44:06,752 DEBUG [c.c.n.NetworkModelImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Service SecurityGroup is not supported in the
>> network id=205
>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) DeploymentPlanner allocation algorithm: null
>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Trying to allocate a host and storage pools from
>> dc:1, pod:null,cluster:null, requested cpu: 8000, requested ram:
>> 8594128896
>> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Is ROOT volume READY (pool already allocated)?: Yes
>> 2018-12-13 08:44:06,783 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Deploy avoids pods: [], clusters: [], hosts: []
>> 2018-12-13 08:44:06,784 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) This VM has last host_id specified, trying to choose
>> the same host: 10
>> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Host: 10 has cpu capability (cpu:12, speed:3000) to
>> support requested CPU: 4 and requested speed: 2000
>> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Checking if host: 10 has enough capacity for
>> requested CPU: 8000 and requested RAM: 8594128896 ,
>> cpuOverprovisioningFactor: 1.0
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Hosts's actual total CPU: 36000 and CPU after
>> applying overprovisioning: 36000
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) We need to allocate to the last host again, so
>> checking if there is enough reserved capacity
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Reserved CPU: 8000 , Requested CPU: 8000
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Reserved RAM: 8594128896 , Requested RAM: 8594128896
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Host has enough CPU and RAM available
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) STATS: Can alloc CPU from host: 10, used: 4500,
>> reserved: 8000, actual total: 36000, total with overprovisioning:
>> 36000; requested cpu:8000,alloc_from_last_host?:true
>> ,considerReservedCapacity?: true
>> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) STATS: Can alloc MEM from host: 10, used:
>> 2415919104, reserved: 8594128896, total: 49554284544; requested mem:
>> 8594128896,alloc_from_last_host?:true ,considerReservedCapacity?: true
>> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) The last host of this VM is UP and has enough capacity
>> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Now checking for suitable pools under zone: 1, pod:
>> 1, cluster: 1
>> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type): (8,ROOT)
>> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>> can be reused, poolId: 1
>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>> since its READY
>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
>> (12,DATADISK)
>> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>> can be reused, poolId: 1
>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>> since its READY
>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
>> (17,DATADISK)
>> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>> can be reused, poolId: 1
>> 2018-12-13 08:44:06,819 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>> since its READY
>> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
>> (58,DATADISK)
>> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
>> can be reused, poolId: 1
>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Planner need not allocate a pool for this volume
>> since its READY
>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Trying to find a potenial host and associated
>> storage pools from the suitable host/pool lists for this VM
>> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>> pool for volume: DATADISK
>> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Host: 10 can access pool: 1
>> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>> pool for volume: DATADISK
>> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Host: 10 can access pool: 1
>> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>> pool for volume: DATADISK
>> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Host: 10 can access pool: 1
>> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
>> pool for volume: ROOT
>> 2018-12-13 08:44:06,831 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Host: 10 can access pool: 1
>> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Found a potential host id: 10 name: cshp143 and
>> associated storage pools for this VM
>> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Returning Deployment Destination:
>> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
>> : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(10)-Storage()]
>> 2018-12-13 08:44:06,856 DEBUG [c.c.u.d.T.Transaction]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Rolling back the transaction: Time = 3 Name =
>> API-Job-Executor-8; called by
>> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:233-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
>> 2018-12-13 08:44:06,861 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Unable to schedule async job for command
>> com.cloud.vm.VmWorkStart, unexpected exception.
>> javax.persistence.EntityExistsException: Entity already exists:
>> at
>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
>> at
>> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
>> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:233)
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>> at
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
>> at
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>> at
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by:
>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>> Duplicate entry '1344' for key 'PRIMARY'
>> at
>> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> ... 59 more
>> 2018-12-13 08:44:06,876 DEBUG [c.c.u.d.T.Transaction]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Rolling back the transaction: Time = 2 Name =
>> API-Job-Executor-8; called by
>> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:245-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
>> 2018-12-13 08:44:06,880 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
>> (logid:6e9a71c5) Unable to schedule async job for command
>> com.cloud.vm.VmWorkStart, unexpected exception.
>> javax.persistence.EntityExistsException: Entity already exists:
>> at
>> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
>> at
>> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
>> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>> at
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
>> at
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>> at
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by:
>> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>> Duplicate entry '1345' for key 'PRIMARY'
>> at
>> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> ... 59 more
>> 2018-12-13 08:44:06,889 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5)
>> Unexpected exception while executing
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
>> com.cloud.utils.exception.CloudRuntimeException: Unable to schedule
>> async job for command com.cloud.vm.VmWorkStart, unexpected exception.
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
>> at
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
>> at
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>> at
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
>> at
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>> at
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> 2018-12-13 08:44:06,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Complete
>> async job-1343, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
>> to schedule async job for command com.cloud.vm.VmWorkStart,
>> unexpected exception."}
>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Publish
>> async job-1343 complete on message bus
>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
>> jobs related to job-1343
>> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Update db
>> status for job-1343
>> 2018-12-13 08:44:06,895 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
>> jobs joined with job-1343 and disjoin all subjobs created from job- 1343
>> 2018-12-13 08:44:06,899 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Done
>> executing
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin for
>> job-1343
>> 2018-12-13 08:44:06,899 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Remove
>> job-1343 from job monitoring
>> 2018-12-13 08:44:07,056 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) Begin cleanup
>> expired async-jobs
>> 2018-12-13 08:44:07,062 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) End cleanup
>> expired async-jobs
>>
>> This problem has already appeared to me other times (in other
>> circumstances) and I have reported it but I have not found a solution.
>>
>> Can you help me identify the problem?
>>
>
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.
Re: URGENT: Unable to schedule async job for command
com.cloud.vm.VmWorkStart, unexpected exception
Posted by Ugo Vasi <ug...@procne.it.INVALID>.
We have verified that the problem is not tied to a vm but it also
concerns the other VMs that are stopped. Trying to restart them gives
the same error message.
Restoring the previous day's database which problems might imply?
Il 13/12/18 09:01, Ugo Vasi ha scritto:
> Hi all,
> I'm trying to reboot a vm after the host it ran on crashed and
> restarted from the HA system. All the VMs running on the rebooted host
> were restarted on other hosts except one.
> In the web interface and using cloudmonkey I get this message:
> "Unable to schedule async job for command com.cloud.vm.VmWorkStart,
> unexpected exception."
>
> In the management-server.log file there would seem to be a problem
> when creating an element that is duplicated (Duplicate entry 'xxxx'
> for key 'PRIMARY'):
>
> 2018-12-13 08:44:06,659 DEBUG [c.c.a.ApiServlet]
> (qtp1096283470-445:ctx-6c065e06) (logid:87edf8d7) ===START===
> 10.80.0.6 -- GET
> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
> 2018-12-13 08:44:06,665 DEBUG [c.c.a.ApiServer]
> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) CIDRs
> from which account 'Acct[26e597f2-b5ca-11e8-a619-c8cbb8cb15cd-admin]'
> is allowed to perform API calls: 0.0.0.0/0,::/0
> 2018-12-13 08:44:06,693 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:7e0c4dc9) Add
> job-1343 into job monitoring
> 2018-12-13 08:44:06,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7) submit
> async job-1343, details: AsyncJobVO {id:1343, userId: 2, accountId: 2,
> instanceType: VirtualMachine, instanceId: 8, cmd:
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null}
> 2018-12-13 08:44:06,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Executing
> AsyncJobVO {id:1343, userId: 2, accountId: 2, instanceType:
> VirtualMachine, instanceId: 8, cmd:
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"6170","id":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","ctxDetails":"{\"interface
> com.cloud.vm.VirtualMachine\":\"dde566b2-ef2c-4f86-a82b-c8286f0c24f7\"}","ctxAccountId":"2","uuid":"dde566b2-ef2c-4f86-a82b-c8286f0c24f7","cmdEventType":"VM.START","_":"1544687047015"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null}
> 2018-12-13 08:44:06,705 DEBUG [c.c.a.ApiServlet]
> (qtp1096283470-445:ctx-6c065e06 ctx-c8b27e4a) (logid:87edf8d7)
> ===END=== 10.80.0.6 -- GET
> command=startVirtualMachine&response=json&id=dde566b2-ef2c-4f86-a82b-c8286f0c24f7&_=1544687047015
> 2018-12-13 08:44:06,745 DEBUG [c.c.n.NetworkModelImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Service SecurityGroup is not supported in the network
> id=205
> 2018-12-13 08:44:06,752 DEBUG [c.c.n.NetworkModelImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Service SecurityGroup is not supported in the network
> id=205
> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) DeploymentPlanner allocation algorithm: null
> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Trying to allocate a host and storage pools from
> dc:1, pod:null,cluster:null, requested cpu: 8000, requested ram:
> 8594128896
> 2018-12-13 08:44:06,774 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Is ROOT volume READY (pool already allocated)?: Yes
> 2018-12-13 08:44:06,783 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Deploy avoids pods: [], clusters: [], hosts: []
> 2018-12-13 08:44:06,784 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) This VM has last host_id specified, trying to choose
> the same host: 10
> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Host: 10 has cpu capability (cpu:12, speed:3000) to
> support requested CPU: 4 and requested speed: 2000
> 2018-12-13 08:44:06,794 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Checking if host: 10 has enough capacity for
> requested CPU: 8000 and requested RAM: 8594128896 ,
> cpuOverprovisioningFactor: 1.0
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Hosts's actual total CPU: 36000 and CPU after
> applying overprovisioning: 36000
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) We need to allocate to the last host again, so
> checking if there is enough reserved capacity
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Reserved CPU: 8000 , Requested CPU: 8000
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Reserved RAM: 8594128896 , Requested RAM: 8594128896
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Host has enough CPU and RAM available
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) STATS: Can alloc CPU from host: 10, used: 4500,
> reserved: 8000, actual total: 36000, total with overprovisioning:
> 36000; requested cpu:8000,alloc_from_last_host?:true
> ,considerReservedCapacity?: true
> 2018-12-13 08:44:06,797 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) STATS: Can alloc MEM from host: 10, used: 2415919104,
> reserved: 8594128896, total: 49554284544; requested mem:
> 8594128896,alloc_from_last_host?:true ,considerReservedCapacity?: true
> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) The last host of this VM is UP and has enough capacity
> 2018-12-13 08:44:06,797 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Now checking for suitable pools under zone: 1, pod:
> 1, cluster: 1
> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type): (8,ROOT)
> 2018-12-13 08:44:06,803 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> can be reused, poolId: 1
> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> since its READY
> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
> (12,DATADISK)
> 2018-12-13 08:44:06,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> can be reused, poolId: 1
> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> since its READY
> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
> (17,DATADISK)
> 2018-12-13 08:44:06,814 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> can be reused, poolId: 1
> 2018-12-13 08:44:06,819 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> since its READY
> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Checking suitable pools for volume (Id, Type):
> (58,DATADISK)
> 2018-12-13 08:44:06,820 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Volume has pool already allocated, checking if pool
> can be reused, poolId: 1
> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Planner need not allocate a pool for this volume
> since its READY
> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Trying to find a potenial host and associated storage
> pools from the suitable host/pool lists for this VM
> 2018-12-13 08:44:06,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> pool for volume: DATADISK
> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Host: 10 can access pool: 1
> 2018-12-13 08:44:06,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> pool for volume: DATADISK
> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Host: 10 can access pool: 1
> 2018-12-13 08:44:06,829 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> pool for volume: DATADISK
> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Host: 10 can access pool: 1
> 2018-12-13 08:44:06,830 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Checking if host: 10 can access any suitable storage
> pool for volume: ROOT
> 2018-12-13 08:44:06,831 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Host: 10 can access pool: 1
> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Found a potential host id: 10 name: cshp143 and
> associated storage pools for this VM
> 2018-12-13 08:44:06,833 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Returning Deployment Destination:
> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
> : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(10)-Storage()]
> 2018-12-13 08:44:06,856 DEBUG [c.c.u.d.T.Transaction]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Rolling back the transaction: Time = 3 Name =
> API-Job-Executor-8; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:233-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
> 2018-12-13 08:44:06,861 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Unable to schedule async job for command
> com.cloud.vm.VmWorkStart, unexpected exception.
> javax.persistence.EntityExistsException: Entity already exists:
> at
> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
> at
> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
> at
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:233)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
> at
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Duplicate entry '1344' for key 'PRIMARY'
> at
> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> ... 59 more
> 2018-12-13 08:44:06,876 DEBUG [c.c.u.d.T.Transaction]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Rolling back the transaction: Time = 2 Name =
> API-Job-Executor-8; called by
> -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.startVmThroughJobQueue:4457-VirtualMachineManagerImpl.advanceStart:899-VirtualMachineManagerImpl.start:718-VMEntityManagerImpl.deployVirtualMachine:245-VirtualMachineEntityImpl.deploy:212-UserVmManagerImpl.startVirtualMachine:4495
> 2018-12-13 08:44:06,880 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343 ctx-eb44a3e5)
> (logid:6e9a71c5) Unable to schedule async job for command
> com.cloud.vm.VmWorkStart, unexpected exception.
> javax.persistence.EntityExistsException: Entity already exists:
> at
> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
> at
> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
> at
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
> at
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Duplicate entry '1345' for key 'PRIMARY'
> at
> sun.reflect.GeneratedConstructorAccessor130.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> ... 59 more
> 2018-12-13 08:44:06,889 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
> com.cloud.utils.exception.CloudRuntimeException: Unable to schedule
> async job for command com.cloud.vm.VmWorkStart, unexpected exception.
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
> at
> com.cloud.vm.VirtualMachineManagerImpl.startVmThroughJobQueue(VirtualMachineManagerImpl.java:4457)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:899)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:718)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4495)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2689)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy164.startVirtualMachine(Unknown Source)
> at
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin.execute(StartVMCmdByAdmin.java:51)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-12-13 08:44:06,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Complete
> async job-1343, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
> to schedule async job for command com.cloud.vm.VmWorkStart, unexpected
> exception."}
> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Publish
> async job-1343 complete on message bus
> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
> jobs related to job-1343
> 2018-12-13 08:44:06,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Update db
> status for job-1343
> 2018-12-13 08:44:06,895 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Wake up
> jobs joined with job-1343 and disjoin all subjobs created from job- 1343
> 2018-12-13 08:44:06,899 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Done
> executing org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
> for job-1343
> 2018-12-13 08:44:06,899 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-8:ctx-b5905c86 job-1343) (logid:6e9a71c5) Remove
> job-1343 from job monitoring
> 2018-12-13 08:44:07,056 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) Begin cleanup
> expired async-jobs
> 2018-12-13 08:44:07,062 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-31fc417b) (logid:36cfc76d) End cleanup
> expired async-jobs
>
> This problem has already appeared to me other times (in other
> circumstances) and I have reported it but I have not found a solution.
>
> Can you help me identify the problem?
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.