[jira] [Created] (CLOUDSTACK-7832) MySQL deadlock occurred in resetting job_executing_msid of the completed vm worker job causing job queue stuck.

["Min Chen (JIRA)" <ji...@apache.org>]

Min Chen created CLOUDSTACK-7832:
------------------------------------

             Summary: MySQL deadlock occurred in resetting job_executing_msid of the completed vm worker job causing job queue stuck.
                 Key: CLOUDSTACK-7832
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7832
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: Management Server
    Affects Versions: 4.3.0
            Reporter: Min Chen
            Assignee: Min Chen
            Priority: Critical
             Fix For: 4.5.0


This is mainly happening for an vm operation involving two VM work jobs consecutively, for example, destroyVM which involved two steps: stopVM and deleteAllVMSnapshots. In updating job_executing_msid column of the first VM worker job, the second VM worker job has been scheduled to run, then we will see such MySQL deadlock exception happening:

2014-09-17 03:26:40,538 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-164:job-831168/job-831176) Double excep
tion
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@42c4a7ea: UPDATE async_
job SET async_job.job_executing_msid=null WHERE async_job.id = 831176
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

Due to this exception, the queue item is not purged for the first vm worker job, then all the following operations for the same VM will get stuck.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)