You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Sanjeev N (JIRA)" <ji...@apache.org> on 2014/02/26 12:12:20 UTC
[jira] [Closed] (CLOUDSTACK-5745) [Hyper-v] System vms live
migration does not work
[ https://issues.apache.org/jira/browse/CLOUDSTACK-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjeev N closed CLOUDSTACK-5745.
---------------------------------
Verified this bug on latest build from 4.3 branch with commit 699798ec278ae52a5bdd0adfa9f5023833e5266f. System vms live migration works fine.
> [Hyper-v] System vms live migration does not work
> -------------------------------------------------
>
> Key: CLOUDSTACK-5745
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5745
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Hypervisor Controller
> Affects Versions: 4.3.0
> Environment: Latest build from 4.3 branch with commit:
> 95b6a7b96dab1c77e25987d6fe6003b4447281f1
> Reporter: Sanjeev N
> Assignee: Devdeep Singh
> Priority: Critical
> Labels: hyper-V,, hyper-v, hyperv
> Fix For: 4.3.0
>
> Attachments: management-server.rar
>
>
> [Hyper-v] System vms live migration does not work
> Steps to Reproduce:
> ================
> 1.Bring up CS in advanced zone with at-least two Hyperv hosts in a cluster
> 2.Deploy couple of guest vms.
> 3.Try live migrating any of the system vms (SSVM,CPVM,VR)
> Result:
> ======
> Live migration of system vms failes with NPE. Guest vms live migration works fine.
> Following are the sequence of events during live migrations:
> 2014-01-03 11:31:05,427 DEBUG [c.c.a.ApiServlet] (catalina-exec-22:ctx-71773ac9) ===START=== 10.146.0.133 -- GET command=migrateSystemVm&hostid=a138433d-0aa6-4711-9559-9738ea7e03fc&virtualmachineid=df65d272-b64f-4b5e-9def-e0b34aa481e6&response=json&sessionkey=48pxYt3vmMxm4yn7pwbEszyG7V4%3D&_=1388728864662
> 2014-01-03 11:31:05,570 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-22:ctx-71773ac9 ctx-0b17c466) submit async job-47, details: AsyncJobVO {id:47, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd, cmdInfo: {"response":"json","sessionkey":"48pxYt3vmMxm4yn7pwbEszyG7V4\u003d","virtualmachineid":"df65d272-b64f-4b5e-9def-e0b34aa481e6","cmdEventType":"VM.MIGRATE","hostid":"a138433d-0aa6-4711-9559-9738ea7e03fc","ctxUserId":"2","httpmethod":"GET","_":"1388728864662","ctxAccountId":"2","ctxStartEventId":"109"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-01-03 11:31:05,572 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-57:ctx-f75b2d46) Add job-47 into job monitoring
> 2014-01-03 11:31:05,572 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-57:ctx-f75b2d46) Executing AsyncJobVO {id:47, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd, cmdInfo: {"response":"json","sessionkey":"48pxYt3vmMxm4yn7pwbEszyG7V4\u003d","virtualmachineid":"df65d272-b64f-4b5e-9def-e0b34aa481e6","cmdEventType":"VM.MIGRATE","hostid":"a138433d-0aa6-4711-9559-9738ea7e03fc","ctxUserId":"2","httpmethod":"GET","_":"1388728864662","ctxAccountId":"2","ctxStartEventId":"109"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-01-03 11:31:05,572 DEBUG [c.c.a.ApiServlet] (catalina-exec-22:ctx-71773ac9 ctx-0b17c466) ===END=== 10.146.0.133 -- GET command=migrateSystemVm&hostid=a138433d-0aa6-4711-9559-9738ea7e03fc&virtualmachineid=df65d272-b64f-4b5e-9def-e0b34aa481e6&response=json&sessionkey=48pxYt3vmMxm4yn7pwbEszyG7V4%3D&_=1388728864662
> 2014-01-03 11:31:05,682 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-57:ctx-f75b2d46 ctx-0b17c466) Sync job-48 execution on object VmWorkJobQueue.4
> 2014-01-03 11:31:05,778 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-3:null) SeqA 4-10358: Processing Seq 4-10358: { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n \"connections\": [\n {\n \"id\": 19,\n \"clientInfo\": \"\",\n \"host\": \"C9FB9529-1A5D-494A-822B-068D485F888B\",\n \"port\": 2179,\n \"tag\": \"df65d272-b64f-4b5e-9def-e0b34aa481e6\",\n \"createTime\": 1388728599282,\n \"lastUsedTime\": 1388728719852\n },\n {\n \"id\": 18,\n \"clientInfo\": \"\",\n \"host\": \"B5FCE673-EA38-43C0-A130-F7FB5E40FF49\",\n \"port\": 2179,\n \"tag\": \"d73f11b3-9fb3-4c4f-aff6-3f02c7e294de\",\n \"createTime\": 1388728576473,\n \"lastUsedTime\": 1388728717138\n },\n {\n \"id\": 17,\n \"clientInfo\": \"\",\n \"host\": \"71F306F9-997C-40E1-8CCE-0611CF71B623\",\n \"port\": 2179,\n \"tag\": \"eaefccc1-6c41-4e57-9911-4dbacac7fe40\",\n \"createTime\": 1388728570588,\n \"lastUsedTime\": 1388728699119\n },\n {\n \"id\": 20,\n \"clientInfo\": \"\",\n \"host\": \"063CC300-DB0C-4CA3-986A-D8E80EC38031\",\n \"port\": 2179,\n \"tag\": \"af3bfdd9-2ca2-4dbd-ba99-e79f2a83319d\",\n \"createTime\": 1388728604238,\n \"lastUsedTime\": 1388728850905\n }\n ]\n}","wait":0}}] }
> 2014-01-03 11:31:05,869 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-3:null) SeqA 4-10358: Sending Seq 4-10358: { Ans: , MgmtId: 132129494109518, via: 4, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2014-01-03 11:31:07,127 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-e32b0652) Execute sync-queue item: SyncQueueItemVO {id:15, queueId: 13, contentType: AsyncJob, contentId: 48, lastProcessMsid: null, lastprocessNumber: null, lastProcessTime: null, created: Fri Jan 03 11:31:05 IST 2014}
> 2014-01-03 11:31:07,128 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-e32b0652) Schedule queued job-48
> 2014-01-03 11:31:07,279 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-58:ctx-de8b3ed7) Add job-48 into job monitoring
> 2014-01-03 11:31:07,279 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-58:ctx-de8b3ed7) Executing AsyncJobVO {id:48, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyABpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAABHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAnNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXEAfgAJcQB-AAlwcQB-AAk, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Jan 03 11:31:05 IST 2014}
> 2014-01-03 11:31:07,279 DEBUG [c.c.v.VmWorkJobDispatcher] (Job-Executor-58:ctx-de8b3ed7) Run VM work job: com.cloud.vm.VmWorkMigrate
> 2014-01-03 11:31:07,281 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Job-Executor-58:ctx-de8b3ed7 ctx-0b17c466) Execute VM work job: com.cloud.vm.VmWorkMigrate{"zoneId":1,"podId":1,"clusterId":1,"hostId":1,"srcHostId":2,"userId":2,"accountId":2,"vmId":4,"handlerName":"VirtualMachineManagerImpl"}
> 2014-01-03 11:31:07,283 ERROR [c.c.v.VmWorkJobHandlerProxy] (Job-Executor-58:ctx-de8b3ed7 ctx-0b17c466) Invocation exception, caused by: java.lang.NullPointerException
> 2014-01-03 11:31:07,283 ERROR [c.c.v.VmWorkJobDispatcher] (Job-Executor-58:ctx-de8b3ed7 ctx-0b17c466) Unable to complete AsyncJobVO {id:48, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyABpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAABHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAnNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXEAfgAJcQB-AAlwcQB-AAk, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Jan 03 11:31:05 IST 2014}
> java.lang.NullPointerException
> at com.cloud.vm.VmWorkMigrate.getDeployDestination(VmWorkMigrate.java:60)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrate(VirtualMachineManagerImpl.java:4758)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4855)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2014-01-03 11:31:07,284 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-58:ctx-de8b3ed7) Unexpected exception
> java.lang.NullPointerException
> 2014-01-03 11:31:07,284 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-58:ctx-de8b3ed7) Complete async job-48, jobStatus: FAILED, resultCode: 530, result: null
> 2014-01-03 11:31:07,379 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-59:ctx-cfa27c50) Add job-47 into job monitoring
> 2014-01-03 11:31:07,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-59:ctx-cfa27c50) Executing AsyncJobVO {id:47, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd, cmdInfo: {"response":"json","sessionkey":"48pxYt3vmMxm4yn7pwbEszyG7V4\u003d","virtualmachineid":"df65d272-b64f-4b5e-9def-e0b34aa481e6","cmdEventType":"VM.MIGRATE","hostid":"a138433d-0aa6-4711-9559-9738ea7e03fc","ctxUserId":"2","httpmethod":"GET","_":"1388728864662","ctxAccountId":"2","ctxStartEventId":"109"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Jan 03 11:31:05 IST 2014}
> 2014-01-03 11:31:07,512 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-59:ctx-cfa27c50) Unable to find a wakeup dispatcher from the joined job: AsyncJobVO {id:47, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd, cmdInfo: {"response":"json","sessionkey":"48pxYt3vmMxm4yn7pwbEszyG7V4\u003d","virtualmachineid":"df65d272-b64f-4b5e-9def-e0b34aa481e6","cmdEventType":"VM.MIGRATE","hostid":"a138433d-0aa6-4711-9559-9738ea7e03fc","ctxUserId":"2","httpmethod":"GET","_":"1388728864662","ctxAccountId":"2","ctxStartEventId":"109"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 132129494109518, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Jan 03 11:31:05 IST 2014}
> 2014-01-03 11:31:07,513 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-59:ctx-cfa27c50) Done executing org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd for job-47
> 2014-01-03 11:31:07,578 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-59:ctx-cfa27c50) Remove job-47 from job monitoring
> 2014-01-03 11:31:07,638 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl] (Job-Executor-58:ctx-de8b3ed7) Sync queue (13) is currently empty
> 2014-01-03 11:31:07,639 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-58:ctx-de8b3ed7) Remove job-48 from job monitoring
> 2014-01-03 11:31:07,688 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-57:ctx-f75b2d46) Complete async job-47, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to migrate the system vm"}
> 2014-01-03 11:31:07,745 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-57:ctx-f75b2d46) Done executing org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd for job-47
> 2014-01-03 11:31:07,778 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-57:ctx-f75b2d46) Remove job-47 from job monitoring
> Also din't see events related to live migration on the agent side. Looks like live migration command has not been passed to the agent.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)