You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Abhinav Roy (JIRA)" <ji...@apache.org> on 2013/12/30 13:48:53 UTC
[jira] [Created] (CLOUDSTACK-5676) Live migration of VM is failing
with a NPE
Abhinav Roy created CLOUDSTACK-5676:
---------------------------------------
Summary: Live migration of VM is failing with a NPE
Key: CLOUDSTACK-5676
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5676
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.3.0
Environment: hyperv , 4.3
Reporter: Abhinav Roy
Priority: Blocker
Fix For: 4.3.0
Steps to Reproduce:
================
1.Bring up CS in advanced zone with 2 hosts in a hyper-v cluster using CIFS for both primary and secondary storage
2.Deploy one or two guest vms using default cent os template.
3.Migrate one of the VMs from Host1 to Host2
Expected Behaviour:
=============
Live VM migration should be successful
Observed Behaviour:
===========
Live VM migration fails with :
013-12-30 16:41:29,564 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-1:ctx-2ba86ed3 ctx-ca0be041) Sync job-95 execution on object VmWorkJobQueue.16
2013-12-30 16:41:31,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-c85fb697) Execute sync-queue item: SyncQueueItemVO {id:30, queueId: 27, contentType: AsyncJob, contentId: 95, lastProcessMsid: null, lastprocessNumber: null, lastProcessTime: null, created: Mon Dec 30 16:41:29 IST 2013}
2013-12-30 16:41:31,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-c85fb697) Schedule queued job-95
2013-12-30 16:41:31,277 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-2:ctx-ba975c56) Add job-95 into job monitoring
2013-12-30 16:41:31,278 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-2:ctx-ba975c56) Executing AsyncJobVO {id:95, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyABpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAEHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXNxAH4ABwAAAAAAAAAEcQB-AAlwcQB-AAk, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null, lastPolled: null, created: Mon Dec 30 16:41:29 IST 2013}
2013-12-30 16:41:31,279 DEBUG [c.c.v.VmWorkJobDispatcher] (Job-Executor-2:ctx-ba975c56) Run VM work job: com.cloud.vm.VmWorkMigrate
2013-12-30 16:41:31,287 ERROR [c.c.v.VmWorkJobDispatcher] (Job-Executor-2:ctx-ba975c56 ctx-ca0be041) Unable to complete AsyncJobVO {id:95, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyABpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAEHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXNxAH4ABwAAAAAAAAAEcQB-AAlwcQB-AAk, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null, lastPolled: null, created: Mon Dec 30 16:41:29 IST 2013}
java.lang.NullPointerException
at com.cloud.vm.VmWorkMigrate.getDeployDestination(VmWorkMigrate.java:60)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4743)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-12-30 16:41:31,291 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-2:ctx-ba975c56) Unexpected exception
java.lang.NullPointerException
at com.google.gson.JsonPrimitive.isPrimitiveOrString(JsonPrimitive.java:316)
at com.google.gson.JsonPrimitive.setValue(JsonPrimitive.java:97)
at com.google.gson.JsonPrimitive.<init>(JsonPrimitive.java:67)
at org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper$ThrowableTypeAdapter.serialize(JobSerializerHelper.java:195)
at org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper$ThrowableTypeAdapter.serialize(JobSerializerHelper.java:158)
at com.google.gson.JsonSerializationVisitor.findAndInvokeCustomSerializer(JsonSerializationVisitor.java:184)
at com.google.gson.JsonSerializationVisitor.visitFieldUsingCustomHandler(JsonSerializationVisitor.java:204)
at com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
at com.google.gson.JsonSerializationContextDefault.serialize(JsonSerializationContextDefault.java:62)
at com.google.gson.JsonSerializationContextDefault.serialize(JsonSerializationContextDefault.java:53)
at com.google.gson.Gson.toJsonTree(Gson.java:220)
at com.google.gson.Gson.toJson(Gson.java:260)
at com.google.gson.Gson.toJson(Gson.java:240)
at org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper.toSerializedString(JobSerializerHelper.java:65)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:105)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-12-30 16:41:31,296 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-2:ctx-ba975c56) Complete async job-95, jobStatus: FAILED, resultCode: 530, result: null
2013-12-30 16:41:31,313 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-3:ctx-71346dc1) Add job-94 into job monitoring
2013-12-30 16:41:31,313 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-3:ctx-71346dc1) Executing AsyncJobVO {id:94, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdInfo: {"response":"json","sessionkey":"lIed4xLaMsDL4yCyAJYyUK0pAJk\u003d","virtualmachineid":"462e6bfc-7c79-4290-9967-8b897ada09e1","cmdEventType":"VM.MIGRATE","hostid":"655f6fb2-82b7-4388-9462-3c15f824e5f9","ctxUserId":"2","httpmethod":"GET","_":"1388401853884","ctxAccountId":"2","ctxStartEventId":"216"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null, lastPolled: null, created: Mon Dec 30 16:41:29 IST 2013}
2013-12-30 16:41:31,328 WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-3:ctx-71346dc1) job-94 is scheduled for wakeup run, but there is no joining info anymore
2013-12-30 16:41:31,333 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-3:ctx-71346dc1) Unable to find a wakeup dispatcher from the joined job: AsyncJobVO {id:94, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdInfo: {"response":"json","sessionkey":"lIed4xLaMsDL4yCyAJYyUK0pAJk\u003d","virtualmachineid":"462e6bfc-7c79-4290-9967-8b897ada09e1","cmdEventType":"VM.MIGRATE","hostid":"655f6fb2-82b7-4388-9462-3c15f824e5f9","ctxUserId":"2","httpmethod":"GET","_":"1388401853884","ctxAccountId":"2","ctxStartEventId":"216"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null, lastPolled: null, created: Mon Dec 30 16:41:29 IST 2013}
2013-12-30 16:41:31,334 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-3:ctx-71346dc1) Done executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-94
2013-12-30 16:41:31,336 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl] (Job-Executor-2:ctx-ba975c56) Sync queue (27) is currently empty
2013-12-30 16:41:31,337 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-2:ctx-ba975c56) Remove job-95 from job monitoring
2013-12-30 16:41:31,347 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-3:ctx-71346dc1) Remove job-94 from job monitoring
2013-12-30 16:41:31,364 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-1:ctx-2ba86ed3 ctx-ca0be041) Complete async job-94, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"462e6bfc-7c79-4290-9967-8b897ada09e1","name":"av8","displayname":"av8","account":"admin","domainid":"3478c004-7129-11e3-89a3-fef34996d384","domain":"ROOT","created":"2013-12-30T16:31:47+0530","state":"Running","haenable":false,"zoneid":"aaa51135-ad9e-4053-9023-17b7923b8694","zonename":"BLR","hostid":"cd56ae63-24c8-4e55-be45-c133f9ecb50f","hostname":"10.102.192.14","templateid":"3483f6ae-7129-11e3-89a3-fef34996d384","templatename":"CentOS 6.4(64-bit) GUI (Hyperv)","templatedisplaytext":"CentOS 6.4 (64-bit) GUI (Hyperv)","passwordenabled":false,"serviceofferingid":"869e9873-ab4c-48c1-a57b-58b73f2939ec","serviceofferingname":"Small Instance","cpunumber":1,"cpuspeed":500,"memory":512,"cpuused":"0%","networkkbsread":1,"networkkbswrite":1,"guestosid":"34a20e96-7129-11e3-89a3-fef34996d384","rootdeviceid":0,"rootdevicetype":"ROOT","securitygroup":[],"nic":[{"id":"e897a7eb-4bea-4335-8925-a240c95df9da","networkid":"80e01c55-75b3-4e7e-9668-fe46ecae4ea8","networkname":"an1","netmask":"255.255.255.0","gateway":"10.1.1.1","ipaddress":"10.1.1.7","isolationuri":"vlan://680","broadcasturi":"vlan://680","traffictype":"Guest","type":"Isolated","isdefault":true,"macaddress":"02:00:08:a2:00:09"}],"hypervisor":"Hyperv","instancename":"i-2-16-VM","tags":[],"affinitygroup":[],"displayvm":true,"isdynamicallyscalable":false}
2013-12-30 16:41:31,372 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-1:ctx-2ba86ed3) Done executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-94
2013-12-30 16:41:31,382 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-2ba86ed3) Remove job-94 from job monitoring
2013-12-30 16:41:32,507 DEBUG [c.c.a.ApiServlet] (catalina-exec-16:ctx-5f391aea) ===START=== 10.144.7.20 -- GET command=queryAsyncJobResult&jobId=f94905d3-a6e7-4119-aa2a-a58c673979c4&response=json&sessionkey=lIed4xLaMsDL4yCyAJYyUK0pAJk%3D&_=1388401856795
Workaround :
=========================
Add a new entry in cloud.configurations table with
vm.job.enabled set to "false"
restart management server. After this Live VM migration works fine.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)