Unable to spawn new System VMs after server restarted

[Tilak Raj Singh <ti...@gmail.com>]

Hi All,

Recently my Server started from backup and last I checked all my Management
IPs were full and hence Cloudstack was not able to spawn new System Vms.
Trying to solv the issue i cleared all the allocated management ips from
op_dc_ip_address_alloc table in database. But now I am getting this issue
of Foreign Key constraint when the management server is trying to expunge a
whose entry was there in the database. As a consequence new system vms are
not able to spawn up. Can somebody help me with details as to which entries
and in which tables should I delete for all this to start working again or
any other help in this regard.

Here is the error log

2015-07-08 20:16:54,880 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Expunging unfinished job AsyncJobVO
{id:15864, userId: 1, accountId: 1, instanceType: null, instanceId: null,
cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAJMHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 91019618482122, completeMsid: null, lastUpdated:
null, lastPolled: null, created: Tue Jun 30 00:59:18 IST 2015}
2015-07-08 20:16:54,882 DEBUG [c.c.u.d.T.Transaction]
(AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Rolling back the transaction: Time =
2 Name =  AsyncJobMgr-Heartbeat-1; called by
-TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-TransactionLegacy.close:667-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy226.expunge:-1-AsyncJobManagerImpl$8.doInTransactionWithoutResult:813-TransactionCallbackNoReturn.doInTransaction:25-Transaction$2.doInTransaction:49
2015-07-08 20:16:54,908 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Unexpected exception when trying to
execute queue item,
com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.PreparedStatement@1ea2e3c8: DELETE FROM async_job WHERE
async_job.id= 15864
at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy226.expunge(Unknown Source)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$8.doInTransactionWithoutResult(AsyncJobManagerImpl.java:813)
at
com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:49)
at com.cloud.utils.db.Transaction.execute(Transaction.java:37)
at com.cloud.utils.db.Transaction.execute(Transaction.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.expungeAsyncJob(AsyncJobManagerImpl.java:810)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.reallyRun(AsyncJobManagerImpl.java:773)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.runInContext(AsyncJobManagerImpl.java:749)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Cannot delete or update a parent row: a foreign key constraint fails
(`cloud`.`async_job_join_map`, CONSTRAINT
`fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
`async_job` (`id`))
at sun.reflect.GeneratedConstructorAccessor116.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
... 32 more

Regards

Re: Unable to spawn new System VMs after server restarted

[Tilak Raj Singh <ti...@gmail.com>]

Hi Somesh,

Sorry for late reply. I ctually deleted all the entries from
op_dc_ip_address alloc table as the cpvm was hung in start phase...the
issue started after that only..I then again entered a dummy entry with
corredponding reservation id and nic id uing nic table and vm_instance
table but still the error is same. Can you guide me which files or db dump
to attach so that you can help me resolve the issue

Regards

Tilak Raj Singh

On Fri, Jul 10, 2015 at 9:51 PM, Somesh Naidu <So...@citrix.com>
wrote:

> Need more logs (the relevant exception stack to be precise) to locate the
> root cause of the error.
>
> In addition, can you check the integrity of the following table/view (just
> perform a select on these):
> async_job_view
> op_dc_ip_address_alloc
>
> Regards,
> Somesh
>
>
> -----Original Message-----
> From: Tilak Raj Singh [mailto:tilak72@gmail.com]
> Sent: Thursday, July 09, 2015 1:36 PM
> To: users@cloudstack.apache.org
> Subject: Re: Unable to spawn new System VMs after server restarted
>
> hi all...can somebody please guide me through this process
>
> Regards
>
> On Wed, Jul 8, 2015 at 8:25 PM, Tilak Raj Singh <ti...@gmail.com> wrote:
>
> > Hi All,
> >
> > Recently my Server started from backup and last I checked all my
> > Management IPs were full and hence Cloudstack was not able to spawn new
> > System Vms.
> > Trying to solv the issue i cleared all the allocated management ips from
> > op_dc_ip_address_alloc table in database. But now I am getting this issue
> > of Foreign Key constraint when the management server is trying to
> expunge a
> > whose entry was there in the database. As a consequence new system vms
> are
> > not able to spawn up. Can somebody help me with details as to which
> entries
> > and in which tables should I delete for all this to start working again
> or
> > any other help in this regard.
> >
> > Here is the error log
> >
> > 2015-07-08 20:16:54,880 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Expunging unfinished job
> AsyncJobVO
> > {id:15864, userId: 1, accountId: 1, instanceType: null, instanceId: null,
> > cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> >
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAJMHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 91019618482122, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: Tue Jun 30 00:59:18 IST 2015}
> > 2015-07-08 20:16:54,882 DEBUG [c.c.u.d.T.Transaction]
> > (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Rolling back the transaction:
> Time =
> > 2 Name =  AsyncJobMgr-Heartbeat-1; called by
> >
> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-TransactionLegacy.close:667-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy226.expunge:-1-AsyncJobManagerImpl$8.doInTransactionWithoutResult:813-TransactionCallbackNoReturn.doInTransaction:25-Transaction$2.doInTransaction:49
> > 2015-07-08 20:16:54,908 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Unexpected exception when trying
> to
> > execute queue item,
> > com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> > com.mysql.jdbc.PreparedStatement@1ea2e3c8: DELETE FROM async_job WHERE
> > async_job.id= 15864
> > at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
> > at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> > at
> >
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> > at
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> > at
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> > at com.sun.proxy.$Proxy226.expunge(Unknown Source)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$8.doInTransactionWithoutResult(AsyncJobManagerImpl.java:813)
> > at
> >
> com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
> > at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:49)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:37)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:46)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.expungeAsyncJob(AsyncJobManagerImpl.java:810)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.reallyRun(AsyncJobManagerImpl.java:773)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.runInContext(AsyncJobManagerImpl.java:749)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by:
> >
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> > Cannot delete or update a parent row: a foreign key constraint fails
> > (`cloud`.`async_job_join_map`, CONSTRAINT
> > `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`)
> REFERENCES
> > `async_job` (`id`))
> > at sun.reflect.GeneratedConstructorAccessor116.newInstance(Unknown
> Source)
> > at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > at com.mysql.jdbc.Util.getInstance(Util.java:386)
> > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
> > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
> > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
> > at
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
> > at
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> > at
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> > at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
> > ... 32 more
> >
> > Regards
> >
>

RE: Unable to spawn new System VMs after server restarted

[Somesh Naidu <So...@citrix.com>]

Need more logs (the relevant exception stack to be precise) to locate the root cause of the error.

In addition, can you check the integrity of the following table/view (just perform a select on these):
async_job_view
op_dc_ip_address_alloc

Regards,
Somesh


-----Original Message-----
From: Tilak Raj Singh [mailto:tilak72@gmail.com] 
Sent: Thursday, July 09, 2015 1:36 PM
To: users@cloudstack.apache.org
Subject: Re: Unable to spawn new System VMs after server restarted

hi all...can somebody please guide me through this process

Regards

On Wed, Jul 8, 2015 at 8:25 PM, Tilak Raj Singh <ti...@gmail.com> wrote:

> Hi All,
>
> Recently my Server started from backup and last I checked all my
> Management IPs were full and hence Cloudstack was not able to spawn new
> System Vms.
> Trying to solv the issue i cleared all the allocated management ips from
> op_dc_ip_address_alloc table in database. But now I am getting this issue
> of Foreign Key constraint when the management server is trying to expunge a
> whose entry was there in the database. As a consequence new system vms are
> not able to spawn up. Can somebody help me with details as to which entries
> and in which tables should I delete for all this to start working again or
> any other help in this regard.
>
> Here is the error log
>
> 2015-07-08 20:16:54,880 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Expunging unfinished job AsyncJobVO
> {id:15864, userId: 1, accountId: 1, instanceType: null, instanceId: null,
> cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAJMHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 91019618482122, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: Tue Jun 30 00:59:18 IST 2015}
> 2015-07-08 20:16:54,882 DEBUG [c.c.u.d.T.Transaction]
> (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Rolling back the transaction: Time =
> 2 Name =  AsyncJobMgr-Heartbeat-1; called by
> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-TransactionLegacy.close:667-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy226.expunge:-1-AsyncJobManagerImpl$8.doInTransactionWithoutResult:813-TransactionCallbackNoReturn.doInTransaction:25-Transaction$2.doInTransaction:49
> 2015-07-08 20:16:54,908 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Unexpected exception when trying to
> execute queue item,
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.PreparedStatement@1ea2e3c8: DELETE FROM async_job WHERE
> async_job.id= 15864
> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
> at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy226.expunge(Unknown Source)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$8.doInTransactionWithoutResult(AsyncJobManagerImpl.java:813)
> at
> com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
> at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:49)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:37)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.expungeAsyncJob(AsyncJobManagerImpl.java:810)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.reallyRun(AsyncJobManagerImpl.java:773)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.runInContext(AsyncJobManagerImpl.java:749)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Cannot delete or update a parent row: a foreign key constraint fails
> (`cloud`.`async_job_join_map`, CONSTRAINT
> `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
> `async_job` (`id`))
> at sun.reflect.GeneratedConstructorAccessor116.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
> ... 32 more
>
> Regards
>

Re: Unable to spawn new System VMs after server restarted

[Tilak Raj Singh <ti...@gmail.com>]

hi all...can somebody please guide me through this process

Regards

On Wed, Jul 8, 2015 at 8:25 PM, Tilak Raj Singh <ti...@gmail.com> wrote:

> Hi All,
>
> Recently my Server started from backup and last I checked all my
> Management IPs were full and hence Cloudstack was not able to spawn new
> System Vms.
> Trying to solv the issue i cleared all the allocated management ips from
> op_dc_ip_address_alloc table in database. But now I am getting this issue
> of Foreign Key constraint when the management server is trying to expunge a
> whose entry was there in the database. As a consequence new system vms are
> not able to spawn up. Can somebody help me with details as to which entries
> and in which tables should I delete for all this to start working again or
> any other help in this regard.
>
> Here is the error log
>
> 2015-07-08 20:16:54,880 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Expunging unfinished job AsyncJobVO
> {id:15864, userId: 1, accountId: 1, instanceType: null, instanceId: null,
> cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAJMHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 91019618482122, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: Tue Jun 30 00:59:18 IST 2015}
> 2015-07-08 20:16:54,882 DEBUG [c.c.u.d.T.Transaction]
> (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Rolling back the transaction: Time =
> 2 Name =  AsyncJobMgr-Heartbeat-1; called by
> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-TransactionLegacy.close:667-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy226.expunge:-1-AsyncJobManagerImpl$8.doInTransactionWithoutResult:813-TransactionCallbackNoReturn.doInTransaction:25-Transaction$2.doInTransaction:49
> 2015-07-08 20:16:54,908 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Unexpected exception when trying to
> execute queue item,
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.PreparedStatement@1ea2e3c8: DELETE FROM async_job WHERE
> async_job.id= 15864
> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
> at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy226.expunge(Unknown Source)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$8.doInTransactionWithoutResult(AsyncJobManagerImpl.java:813)
> at
> com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
> at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:49)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:37)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.expungeAsyncJob(AsyncJobManagerImpl.java:810)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.reallyRun(AsyncJobManagerImpl.java:773)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.runInContext(AsyncJobManagerImpl.java:749)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Cannot delete or update a parent row: a foreign key constraint fails
> (`cloud`.`async_job_join_map`, CONSTRAINT
> `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
> `async_job` (`id`))
> at sun.reflect.GeneratedConstructorAccessor116.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
> ... 32 more
>
> Regards
>